From 683b3f322a3277ef422922671efe804b2cce7385 Mon Sep 17 00:00:00 2001 From: Garand Tyson Date: Tue, 10 Nov 2026 10:41:05 -0800 Subject: [PATCH 1/2] Replace SCP nomination with a deterministic leader schedule (prototype) Experimental. Not protocol-gated and not compatible with nominating nodes. Nomination is removed. For slot N every node orders the transitive quorum by SHA256(hash of ledger header N-1 || node id); the leader of round r (a round is a ballot counter, starting at 1) is entry (r - 1) mod n. The round leader builds a value and proposes it straight into the ballot protocol as PREPARE at counter r. Other nodes vote for the leader's ballot at that counter if it passes proposal validation (the former nomination-time checks, e.g. only upgrades the node armed). A leader timer (while the node has no ballot for its round) or the ballot timer moves a node to the next round. Accept, confirm and commit are unchanged, and every new vote still goes through bumpState (value override, updateCurrentValue). Details: - Followers vote for the round 1 leader as soon as they see its ballot, without waiting for their own trigger. - A node that has not voted yet votes for an earlier round's leader ballot at that ballot's counter when it arrives late, and keeps waiting for its current round's leader. - The leader timer gets one extension when the Herder already holds the leader's statement (e.g. it is fetching its transaction set). - A leader's empty-tx-set ballot is followed under ballot validation. - No leader and no leader timer at the UINT32_MAX counter. - State restore sets the value override and the round. - Herder: the trigger starts the slot; the early tx-set build and the follower signature prewarm key off the round leader; NOMINATE envelopes are discarded. New metrics scp.timeout.leader and scp.leader.round; scp.timing.nominated, scp.timeout.nominate and scp.nomination.combinecandidates are gone. Removed: NominationProtocol, the nomination API of SCP/Slot/SCPDriver, combineCandidates, node weights, upgrade stripping, the nomination emit delay, and the nomination-only tests (SCPUnitTests.cpp, nomination cases of SCPTests.cpp, weight / win-rate / timeout statistics in HerderTests.cpp). FORCE_OLD_STYLE_LEADER_ELECTION, VALIDATOR_WEIGHT_CONFIG and the nomination timeout settings still parse but are unused. Tests: leader round unit cases and a randomized multi-node model in SCPTests.cpp, LeaderScheduleTests.cpp, LeaderRoundsTests.cpp (network simulations, including failover with a validator down); Herder, quorum tracker, PendingEnvelopes and overlay IPC tests ported to leader rounds. Known limitations: symmetric, static quorums are assumed (the schedule's node set is each node's view of the transitive quorum); uniform leader weights; the header hash can be ground by a leader; no upgrade stripping or merging; a leader re-proposes a value that is only structurally valid. --- src/herder/Herder.h | 4 +- src/herder/HerderImpl.cpp | 72 +- src/herder/HerderImpl.h | 10 +- src/herder/HerderSCPDriver.cpp | 664 ++---- src/herder/HerderSCPDriver.h | 120 +- src/herder/LeaderSchedule.cpp | 38 + src/herder/LeaderSchedule.h | 26 + src/herder/PendingEnvelopes.cpp | 27 + src/herder/PendingEnvelopes.h | 12 + src/herder/test/HerderTests.cpp | 2009 +++--------------- src/herder/test/LeaderRoundsTests.cpp | 221 ++ src/herder/test/LeaderScheduleTests.cpp | 123 ++ src/herder/test/PendingEnvelopesTests.cpp | 62 +- src/herder/test/QuorumTrackerTests.cpp | 36 +- src/main/Config.cpp | 14 +- src/main/Config.h | 6 +- src/overlay/test/OverlayIPCTests.cpp | 50 +- src/scp/BallotProtocol.cpp | 378 +++- src/scp/BallotProtocol.h | 56 +- src/scp/NominationProtocol.cpp | 926 --------- src/scp/NominationProtocol.h | 152 -- src/scp/SCP.cpp | 40 +- src/scp/SCP.h | 27 +- src/scp/SCPDriver.cpp | 108 - src/scp/SCPDriver.h | 120 +- src/scp/Slot.cpp | 109 +- src/scp/Slot.h | 32 +- src/scp/test/SCPTests.cpp | 2256 +++++++++++---------- src/scp/test/SCPUnitTests.cpp | 840 -------- src/simulation/ApplyLoad.cpp | 11 +- src/test/TxTests.cpp | 2 +- 31 files changed, 2849 insertions(+), 5702 deletions(-) create mode 100644 src/herder/LeaderSchedule.cpp create mode 100644 src/herder/LeaderSchedule.h create mode 100644 src/herder/test/LeaderRoundsTests.cpp create mode 100644 src/herder/test/LeaderScheduleTests.cpp delete mode 100644 src/scp/NominationProtocol.cpp delete mode 100644 src/scp/NominationProtocol.h delete mode 100644 src/scp/test/SCPUnitTests.cpp diff --git a/src/herder/Herder.h b/src/herder/Herder.h index 66207170ee..4492254967 100644 --- a/src/herder/Herder.h +++ b/src/herder/Herder.h @@ -200,8 +200,8 @@ class Herder virtual uint32 getMinLedgerSeqToAskPeers() const = 0; virtual uint32 getMinLedgerSeqToRemember() const = 0; - virtual bool isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt) = 0; + virtual bool isNewerStatement(SCPStatement const& oldSt, + SCPStatement const& newSt) = 0; // Returns sequence number for most recent completed checkpoint that the // node knows about, as derived from diff --git a/src/herder/HerderImpl.cpp b/src/herder/HerderImpl.cpp index e261256430..1a66412bb7 100644 --- a/src/herder/HerderImpl.cpp +++ b/src/herder/HerderImpl.cpp @@ -999,6 +999,12 @@ HerderImpl::recvSCPEnvelope(SCPEnvelope const& envelope) mSCPMetrics.mEnvelopeReceive.Mark(); + if (envelope.statement.pledges.type() == SCP_ST_NOMINATE) + { + // There is no nomination phase (see HerderSCPDriver::getRoundLeader). + return Herder::ENVELOPE_STATUS_DISCARDED; + } + // **** first perform checks that do NOT require signature verification // this allows to fast fail messages that we'd throw away anyways @@ -1691,9 +1697,7 @@ HerderImpl::setupTriggerNextLedger() // trigger. if (getSCP().isValidator() && !triggerDue) { - auto leaders = getSCP().predictNominationLeaders( - nextIndex, xdr::xdr_to_opaque(lcl.header.scpValue), 1); - if (leaders.count(getSCP().getLocalNodeID())) + if (mHerderSCPDriver.isLocalNodeRoundLeader(nextIndex, 1)) { // Validate against the close time we expect at the trigger, // not the earlier wall-clock time at which we start work. @@ -1865,12 +1869,8 @@ HerderImpl::prepareTxSet(uint32_t ledgerSeq, ConsensusTime closeTime) { return; } - auto const& lcl = mLedgerManager.getLastClosedLedgerHeader(); if (!getSCP().isValidator() || - !getSCP() - .predictNominationLeaders( - ledgerSeq, xdr::xdr_to_opaque(lcl.header.scpValue), 1) - .count(getSCP().getLocalNodeID())) + !mHerderSCPDriver.isLocalNodeRoundLeader(ledgerSeq, 1)) { return; } @@ -1889,18 +1889,14 @@ HerderImpl::buildTxSet(uint32_t ledgerSeq, ConsensusTime closeTime) auto const lcl = mLedgerManager.getLastClosedLedgerHeader(); releaseAssert(ledgerSeq == lcl.header.ledgerSeq + 1); releaseAssert(closeTime > getConsensusTime(lcl.header.scpValue)); - bool const activeLeader = getSCP().getNominationLeaders(ledgerSeq).count( - getSCP().getLocalNodeID()); + uint32_t const round = getSCP().getLeaderRound(ledgerSeq); + bool const activeLeader = + round != 0 && mHerderSCPDriver.isLocalNodeRoundLeader(ledgerSeq, round); bool const legacyTxSet = buildsLegacyTxSet(lcl.header.ledgerVersion); - releaseAssert(activeLeader || legacyTxSet || - getSCP() - .predictNominationLeaders( - ledgerSeq, xdr::xdr_to_opaque(lcl.header.scpValue), 1) - .count(getSCP().getLocalNodeID())); CLOG_INFO(Herder, "Building TX set as {} for ledger {}", - activeLeader ? "active nomination leader" - : legacyTxSet ? "validator (legacy tx set)" - : "first-round nomination leader", + activeLeader ? fmt::format("round {} leader", round) + : legacyTxSet ? std::string("validator (legacy tx set)") + : std::string("first-round leader (early)"), ledgerSeq); // Apply-time offset: see the comment in triggerNextLedger auto const closeTimeOffset = @@ -2162,29 +2158,30 @@ HerderImpl::triggerNextLedger(uint32_t ledgerSeqToTrigger, if (!isCtValid) { CLOG_WARNING(Herder, - "Invalid close time selected ({}), skipping nomination", + "Invalid close time selected ({}), skipping the slot", nextCloseTime.toString()); return; } // Start the trigger-to-ballot interval only once this node is going to - // nominate. A skipped trigger must not credit the time spent waiting for + // take part. A skipped trigger must not credit the time spent waiting for // peers to the drift allowance. - mHerderSCPDriver.recordNominationTrigger(ledgerSeqToTrigger); + mHerderSCPDriver.recordTrigger(ledgerSeqToTrigger); if (!getSCP().isValidator()) { - CLOG_DEBUG(Herder, "Non-validating node, skipping nomination (SCP)."); + CLOG_DEBUG(Herder, "Non-validating node, not starting the slot (SCP)."); return; } - getHerderSCPDriver().recordSCPEvent(ledgerSeqToTrigger, true); - mHerderSCPDriver.nominate( + // There is no nomination: the leader of each round (a deterministic + // schedule seeded by the LCL hash) proposes straight into the ballot + // protocol. The value is only built if this node leads a round. + mHerderSCPDriver.startSlot( ledgerSeqToTrigger, [this, ledgerSeqToTrigger]() { - return makeNominationValue(ledgerSeqToTrigger); + return makeProposalValue(ledgerSeqToTrigger); }, - lcl.header.scpValue, /*constructValueNow=*/buildsLegacyTxSet(lcl.header.ledgerVersion)); // A follower validates the leader's set when it arrives, and most of its @@ -2194,9 +2191,10 @@ HerderImpl::triggerNextLedger(uint32_t ledgerSeqToTrigger, // so that validation finds them in the cache. Skip it if this node already // voted, i.e. validated the leader's value. if (!buildsLegacyTxSet(lcl.header.ledgerVersion) && - !getSCP() - .getNominationLeaders(ledgerSeqToTrigger) - .count(getSCP().getLocalNodeID()) && + !mHerderSCPDriver.isLocalNodeRoundLeader( + ledgerSeqToTrigger, + std::max(1, + getSCP().getLeaderRound(ledgerSeqToTrigger))) && getSCP().getLatestMessagesSend(ledgerSeqToTrigger).empty()) { prewarmSignatureCache(ledgerSeqToTrigger, {}); @@ -2444,7 +2442,7 @@ HerderImpl::buildsLegacyTxSet(uint32_t lclLedgerVersion) } ValueWrapperPtr -HerderImpl::makeNominationValue(uint32_t ledgerSeqToTrigger) +HerderImpl::makeProposalValue(uint32_t ledgerSeqToTrigger) { // A follower can be promoted long after the trigger. Recheck the ledger // state and select a fresh close time when SCP actually needs our value. @@ -2522,9 +2520,9 @@ HerderImpl::makeNominationValue(uint32_t ledgerSeqToTrigger) closeTimeOffset); #ifdef BUILD_TESTS - if (mNominatedTxSetOverride) + if (mProposedTxSetOverride) { - proposedSet = mNominatedTxSetOverride(lcl); + proposedSet = mProposedTxSetOverride(lcl); CLOG_INFO(Herder, "Nominating overridden tx set {} for slot {}", hexAbbrev(proposedSet->getContentsHash()), lcl.header.ledgerSeq + 1); @@ -2591,12 +2589,12 @@ HerderImpl::makeNominationValue(uint32_t ledgerSeqToTrigger) } #ifdef BUILD_TESTS - if (mApp.getConfig().TESTING_NOMINATE_RANDOM_VALUES && + if (mApp.getConfig().TESTING_PROPOSE_RANDOM_VALUES && getHerderSCPDriver().protocolAllowsEmptyTxSetValues()) { txSetHash = HashUtils::pseudoRandomForTesting(); CLOG_INFO(Herder, - "TESTING_NOMINATE_RANDOM_VALUES: nominating slot {} " + "TESTING_PROPOSE_RANDOM_VALUES: proposing slot {} " "with random tx-set hash {}", slotIndex, hexAbbrev(txSetHash)); } @@ -3623,9 +3621,9 @@ HerderImpl::makeStellarValue(Hash const& txSetHash, ConsensusTime closeTime, } bool -HerderImpl::isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt) +HerderImpl::isNewerStatement(SCPStatement const& oldSt, + SCPStatement const& newSt) { - return getSCP().isNewerNominationOrBallotSt(oldSt, newSt); + return getSCP().isNewerStatement(oldSt, newSt); } } diff --git a/src/herder/HerderImpl.h b/src/herder/HerderImpl.h index fb29ba924b..f46ad8bf7d 100644 --- a/src/herder/HerderImpl.h +++ b/src/herder/HerderImpl.h @@ -138,7 +138,7 @@ class HerderImpl : public Herder // still run through their production paths. std::function(size_t)> mGetTopTransactionsForTesting; - friend class EarlyNominationTestAccess; + friend class LeaderBuildTestAccess; std::optional mMaxClassicTxSize; std::optional mMaxTxSizeOverride; @@ -162,7 +162,7 @@ class HerderImpl : public Herder // When set, this node nominates the tx set returned by this function // (given the LCL) instead of the one it builds. std::function - mNominatedTxSetOverride; + mProposedTxSetOverride; #endif std::vector getSCPStateForPeer(uint32 ledgerSeq) override; @@ -195,8 +195,8 @@ class HerderImpl : public Herder uint32_t getMinLedgerSeqToRemember() const override; - bool isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt) override; + bool isNewerStatement(SCPStatement const& oldSt, + SCPStatement const& newSt) override; uint32_t getMostRecentCheckpointSeq() override; @@ -272,7 +272,7 @@ class HerderImpl : public Herder PreparedTxSet buildTxSet(uint32_t ledgerSeq, ConsensusTime closeTime); void prepareTxSet(uint32_t ledgerSeq, ConsensusTime closeTime); void discardPreparedTxSet(); - ValueWrapperPtr makeNominationValue(uint32_t ledgerSeq); + ValueWrapperPtr makeProposalValue(uint32_t ledgerSeq); // Whether a ledger closing on an LCL with this protocol uses a legacy // (non-generalized) tx set, which every validator builds locally. static bool buildsLegacyTxSet(uint32_t lclLedgerVersion); diff --git a/src/herder/HerderSCPDriver.cpp b/src/herder/HerderSCPDriver.cpp index 84a2c58820..4a04b8217f 100644 --- a/src/herder/HerderSCPDriver.cpp +++ b/src/herder/HerderSCPDriver.cpp @@ -8,6 +8,7 @@ #include "crypto/SHA.h" #include "crypto/SecretKey.h" #include "herder/HerderImpl.h" +#include "herder/LeaderSchedule.h" #include "herder/LedgerCloseData.h" #include "herder/PendingEnvelopes.h" #include "ledger/LedgerHeaderUtils.h" @@ -54,10 +55,6 @@ HerderSCPDriver::SCPMetrics::SCPMetrics(Application& app) , mValueValid(app.getMetrics().NewMeter({"scp", "value", "valid"}, "value")) , mValueInvalid( app.getMetrics().NewMeter({"scp", "value", "invalid"}, "value")) - , mCombinedCandidates(app.getMetrics().NewMeter( - {"scp", "nomination", "combinecandidates"}, "value")) - , mNominateToPrepare( - app.getMetrics().NewTimer({"scp", "timing", "nominated"})) , mPrepareToExternalize( app.getMetrics().NewTimer({"scp", "timing", "externalized"})) , mTriggerToBallot( @@ -88,13 +85,14 @@ HerderSCPDriver::HerderSCPDriver(Application& app, HerderImpl& herder, , mSCP{*this, mApp.getConfig().NODE_SEED.getPublicKey(), mApp.getConfig().NODE_IS_VALIDATOR, mApp.getConfig().QUORUM_SET} , mSCPMetrics{mApp} - , mNominateTimeout{mApp.getMetrics().NewHistogram( - {"scp", "timeout", "nominate"})} + , mLeaderTimeout{mApp.getMetrics().NewHistogram( + {"scp", "timeout", "leader"})} + , mLeaderRound{mApp.getMetrics().NewHistogram({"scp", "leader", "round"})} , mPrepareTimeout{mApp.getMetrics().NewHistogram( {"scp", "timeout", "prepare"})} , mUniqueValues{mApp.getMetrics().NewHistogram( {"scp", "slot", "values-referenced"})} - , mLedgerSeqNominating(0) + , mLedgerSeqProposing(0) , mTxSetValidCache(TXSETVALID_CACHE_SIZE) { } @@ -406,7 +404,7 @@ HerderSCPDriver::validatePastOrFutureValue( SCPDriver::ValidationLevel HerderSCPDriver::validateValueAgainstLocalState(uint64_t slotIndex, StellarValue const& b, - bool nomination) const + bool proposal) const { ZoneScoped; releaseAssert(threadIsMain()); @@ -437,13 +435,13 @@ HerderSCPDriver::validateValueAgainstLocalState(uint64_t slotIndex, return SCPDriver::kInvalidValue; } - if (nomination) + if (proposal) { - // Empty-tx-set values should only appear in balloting, and so - // are considered invalid during nomination. + // Empty-tx-set values only come out of balloting (a node + // gives up on a download): nobody proposes one. CLOG_DEBUG(Herder, "HerderSCPDriver::validateValue i: {} rejecting " - "empty-tx-set value during nomination", + "empty-tx-set value as a proposal", slotIndex); return SCPDriver::kInvalidValue; } @@ -609,14 +607,14 @@ HerderSCPDriver::deserializeAndValidateStellarValue(uint64_t slotIndex, } void -HerderSCPDriver::extractValidUpgrades(StellarValue& sv, bool nomination) const +HerderSCPDriver::extractValidUpgrades(StellarValue& sv, bool proposal) const { LedgerUpgradeType lastUpgradeType = LEDGER_UPGRADE_VERSION; LedgerUpgradeType thisUpgradeType; bool first = true; for (auto it = sv.upgrades.begin(); it != sv.upgrades.end();) { - if (!mUpgrades.isValid(*it, thisUpgradeType, nomination, mApp)) + if (!mUpgrades.isValid(*it, thisUpgradeType, proposal, mApp)) { it = sv.upgrades.erase(it); } @@ -635,7 +633,7 @@ HerderSCPDriver::extractValidUpgrades(StellarValue& sv, bool nomination) const SCPDriver::ValidationLevel HerderSCPDriver::validateValue(uint64_t slotIndex, Value const& value, - bool nomination) const + bool proposal) const { ZoneScoped; releaseAssert(threadIsMain()); @@ -648,11 +646,11 @@ HerderSCPDriver::validateValue(uint64_t slotIndex, Value const& value, } SCPDriver::ValidationLevel res = - validateValueAgainstLocalState(slotIndex, b, nomination); + validateValueAgainstLocalState(slotIndex, b, proposal); if (res != SCPDriver::kInvalidValue) { auto origSize = b.upgrades.size(); - extractValidUpgrades(b, nomination); + extractValidUpgrades(b, proposal); if (b.upgrades.size() != origSize) { CLOG_TRACE(Herder, @@ -674,27 +672,6 @@ HerderSCPDriver::validateValue(uint64_t slotIndex, Value const& value, return res; } -ValueWrapperPtr -HerderSCPDriver::extractValidValue(uint64_t slotIndex, Value const& value) -{ - ZoneScoped; - StellarValue b; - if (!deserializeAndValidateStellarValue(slotIndex, value, b)) - { - return nullptr; - } - - ValueWrapperPtr res; - if (validateValueAgainstLocalState(slotIndex, b, true) >= - SCPDriver::kStructurallyValidValue) - { - extractValidUpgrades(b, true); - res = wrapStellarValue(b); - } - - return res; -} - // value marshaling std::string @@ -776,18 +753,6 @@ void HerderSCPDriver::timerCallbackWrapper(uint64_t slotIndex, int timerID, std::function cb) { -#ifdef BUILD_TESTS - if (timerID == Slot::NOMINATION_EMIT_TIMER) - { - if (!mHerder.isTracking() || - mHerder.nextConsensusLedgerIndex() == slotIndex) - { - cb(); - } - return; - } -#endif - // reschedule timers for future slots when tracking if (mHerder.isTracking() && mHerder.nextConsensusLedgerIndex() != slotIndex) { @@ -811,11 +776,8 @@ HerderSCPDriver::timerCallbackWrapper(uint64_t slotIndex, int timerID, } else { - if (!SCPTiming.mPrepareStart) - { - // Timeout happened between nominate and first prepare - ++SCPTiming.mNominationTimeoutCount; - } + // Timed out waiting for the round leader's proposal + ++SCPTiming.mLeaderTimeoutCount; } } @@ -876,7 +838,7 @@ HerderSCPDriver::stopTimer(uint64 slotIndex, int timerID) static uint32_t const MAX_TIMEOUT_MS = (30 * 60) * 1000; std::chrono::milliseconds -HerderSCPDriver::computeTimeout(uint32 roundNumber, bool isNomination) +HerderSCPDriver::computeTimeout(uint32 roundNumber) { releaseAssertOrThrow(roundNumber > 0); @@ -891,18 +853,8 @@ HerderSCPDriver::computeTimeout(uint32 roundNumber, bool isNomination) { auto const& networkConfig = mLedgerManager.getLastClosedSorobanNetworkConfig(); - if (isNomination) - { - initialTimeoutMS = - networkConfig.nominationTimeoutInitialMilliseconds(); - incrementMS = - networkConfig.nominationTimeoutIncrementMilliseconds(); - } - else - { - initialTimeoutMS = networkConfig.ballotTimeoutInitialMilliseconds(); - incrementMS = networkConfig.ballotTimeoutIncrementMilliseconds(); - } + initialTimeoutMS = networkConfig.ballotTimeoutInitialMilliseconds(); + incrementMS = networkConfig.ballotTimeoutIncrementMilliseconds(); } auto timeoutMS = initialTimeoutMS + (roundNumber - 1) * incrementMS; @@ -913,284 +865,6 @@ HerderSCPDriver::computeTimeout(uint32 roundNumber, bool isNomination) return std::chrono::milliseconds(timeoutMS); } -#ifdef BUILD_TESTS -std::chrono::milliseconds -HerderSCPDriver::getNominationEmitDelayForTesting() const -{ - return mApp.getConfig().ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING; -} -#endif - -// returns true if l < r -// lh, rh are the hashes of l,h -static bool -compareTxSets(ApplicableTxSetFrameConstPtr const& l, - ApplicableTxSetFrameConstPtr const& r, Hash const& lh, - Hash const& rh, std::optional lEncodedSize, - std::optional rEncodedSize, LedgerHeader const& header, - Hash const& s) -{ - if (!l && !r) - { - // Do not have either tx set. Compare hashes - return lessThanXored(lh, rh, s); - } - - if (!l || !r) - { - // If one exists, choose it - return !l; - } - - auto lSize = l->size(header); - auto rSize = r->size(header); - if (lSize != rSize) - { - return lSize < rSize; - } - if (protocolVersionStartsFrom(header.ledgerVersion, - SOROBAN_PROTOCOL_VERSION)) - { - auto lBids = l->getTotalInclusionFees(); - auto rBids = r->getTotalInclusionFees(); - if (lBids != rBids) - { - return lBids < rBids; - } - } - if (protocolVersionStartsFrom(header.ledgerVersion, ProtocolVersion::V_11)) - { - auto lFee = l->getTotalFees(header); - auto rFee = r->getTotalFees(header); - if (lFee != rFee) - { - return lFee < rFee; - } - } - if (protocolVersionStartsFrom(header.ledgerVersion, - SOROBAN_PROTOCOL_VERSION)) - { - if (lEncodedSize.value() != rEncodedSize.value()) - { - // Look for the smallest encoded size. - return lEncodedSize.value() > rEncodedSize.value(); - } - } - return lessThanXored(lh, rh, s); -} - -ValueWrapperPtr -HerderSCPDriver::combineCandidates(uint64_t slotIndex, - ValueWrapperPtrSet const& candidates) -{ - ZoneScoped; - CLOG_DEBUG(Herder, "Combining {} candidates", candidates.size()); - mSCPMetrics.mCombinedCandidates.Mark(candidates.size()); - - std::map upgrades; - - std::set aggSet; - - releaseAssert(!mLedgerManager.isApplying()); - releaseAssert(threadIsMain()); - auto const& lcl = mLedgerManager.getLastClosedLedgerHeader(); - - Hash candidatesHash; - - std::vector candidateValues; - - for (auto const& c : candidates) - { - candidateValues.emplace_back(); - StellarValue& sv = candidateValues.back(); - Value const& val = c->getValue(); - uint256 const valHash = sha256(val); - - if (!toStellarValue(val, sv)) - { - throw std::runtime_error(fmt::format( - "HerderSCPDriver::combineCandidates: cannot parse candidate " - "value with hash {}", - binToHex(valHash))); - } - - candidatesHash ^= valHash; - - for (auto const& upgrade : sv.upgrades) - { - LedgerUpgrade lupgrade; - try - { - xdr::xdr_from_opaque(upgrade, lupgrade); - } - catch (...) - { - throw std::runtime_error( - fmt::format("HerderSCPDriver::combineCandidates: cannot " - "parse upgrade in candidate with hash {}", - binToHex(valHash))); - } - auto it = upgrades.find(lupgrade.type()); - if (it == upgrades.end()) - { - upgrades.emplace(std::make_pair(lupgrade.type(), lupgrade)); - } - else - { - LedgerUpgrade& clUpgrade = it->second; - switch (lupgrade.type()) - { - case LEDGER_UPGRADE_VERSION: - // pick the highest version - clUpgrade.newLedgerVersion() = - std::max(clUpgrade.newLedgerVersion(), - lupgrade.newLedgerVersion()); - break; - case LEDGER_UPGRADE_BASE_FEE: - // take the max fee - clUpgrade.newBaseFee() = - std::max(clUpgrade.newBaseFee(), lupgrade.newBaseFee()); - break; - case LEDGER_UPGRADE_MAX_TX_SET_SIZE: - // take the max tx set size - clUpgrade.newMaxTxSetSize() = - std::max(clUpgrade.newMaxTxSetSize(), - lupgrade.newMaxTxSetSize()); - break; - case LEDGER_UPGRADE_BASE_RESERVE: - // take the max base reserve - clUpgrade.newBaseReserve() = std::max( - clUpgrade.newBaseReserve(), lupgrade.newBaseReserve()); - break; - case LEDGER_UPGRADE_FLAGS: - clUpgrade.newFlags() = - std::max(clUpgrade.newFlags(), lupgrade.newFlags()); - break; - case LEDGER_UPGRADE_CONFIG: - if (clUpgrade.newConfig().contractID < - lupgrade.newConfig().contractID) - { - clUpgrade.newConfig() = lupgrade.newConfig(); - } - else if (clUpgrade.newConfig().contractID == - lupgrade.newConfig().contractID && - clUpgrade.newConfig().contentHash < - lupgrade.newConfig().contentHash) - { - clUpgrade.newConfig() = lupgrade.newConfig(); - } - break; - case LEDGER_UPGRADE_MAX_SOROBAN_TX_SET_SIZE: - clUpgrade.newMaxSorobanTxSetSize() = - std::max(clUpgrade.newMaxSorobanTxSetSize(), - lupgrade.newMaxSorobanTxSetSize()); - break; - default: - // should never get there with values that are not valid - throw std::runtime_error("invalid upgrade step"); - } - } - } - } - - StellarValue comp; - // take the txSet with the biggest size, highest xored hash that we have - { - auto highest = candidateValues.cend(); - TxSetXDRFrameConstPtr highestTxSet; - ApplicableTxSetFrameConstPtr highestApplicableTxSet; - for (auto it = candidateValues.cbegin(); it != candidateValues.cend(); - ++it) - { - auto const& sv = *it; - TxSetXDRFrameConstPtr cTxSet; - auto const cTxSetResult = mPendingEnvelopes.getTxSet(sv.txSetHash); - if (auto const* ptr = - std::get_if(&cTxSetResult)) - { - cTxSet = *ptr; - } - // else: EmptyTxSet -> cTxSet stays null, handled by existing - - // Prefer applicable tx sets. `compareTxSets` down-ranks - // non-applicable ones. Without CAP-0083 every candidate here is - // applicable (a mismatched one can't be ratified), so the - // non-applicable case doesn't arise. Under CAP-0083 a - // non-applicable candidate may be selected and is replaced with an - // empty-tx-set value during balloting. - ApplicableTxSetFrameConstPtr cApplicableTxSet = nullptr; - if (cTxSet && cTxSet->previousLedgerHash() == lcl.hash) - { - cApplicableTxSet = cTxSet->prepareForApply(mApp, lcl.header); - } - - if (highest == candidateValues.cend() || - compareTxSets(highestApplicableTxSet, cApplicableTxSet, - highest->txSetHash, sv.txSetHash, - highestTxSet ? std::make_optional( - highestTxSet->encodedSize()) - : std::nullopt, - cTxSet ? std::make_optional(cTxSet->encodedSize()) - : std::nullopt, - lcl.header, candidatesHash)) - { - highest = it; - highestTxSet = cTxSet; - highestApplicableTxSet = std::move(cApplicableTxSet); - } - } - if (highest == candidateValues.cend()) - { - throw std::runtime_error( - "No highest candidate transaction set found"); - } - comp = *highest; - } - comp.upgrades.clear(); - for (auto const& upgrade : upgrades) - { - Value v(xdr::xdr_to_opaque(upgrade.second)); - comp.upgrades.emplace_back(v.begin(), v.end()); - } - - auto res = wrapStellarValue(comp); - return res; -} - -bool -HerderSCPDriver::hasUpgrades(Value const& v) -{ - StellarValue sv; - if (!toStellarValue(v, sv)) - { - return false; - } - return !sv.upgrades.empty(); -} - -ValueWrapperPtr -HerderSCPDriver::stripAllUpgrades(Value const& v) -{ - StellarValue sv; - if (!toStellarValue(v, sv)) - { - return nullptr; - } - - // Remove all upgrades - sv.upgrades.clear(); - - // Serialize back to Value - return wrapStellarValue(sv); -} - -uint32_t -HerderSCPDriver::getUpgradeNominationTimeoutLimit() const -{ - return mUpgrades.getParameters().mNominationTimeoutLimit.value_or( - std::numeric_limits::max()); -} - std::optional HerderSCPDriver::getTxSetDownloadWaitTime(Value const& v) const { @@ -1253,17 +927,11 @@ HerderSCPDriver::valueExternalized(uint64_t slotIndex, Value const& value) logQuorumInformationAndUpdateMetrics(slotIndex - 2); } - if (mLedgerSeqNominating != 0) - { - // stop nomination - // this may or may not be the ledger that is currently externalizing - // in both cases, we want to stop nomination as: - // either we're closing the current ledger (typical case) - // or we're going to trigger catchup from history - mSCP.stopNomination(mLedgerSeqNominating); - mCurrentValue.reset(); - mLedgerSeqNominating = 0; - } + // this may or may not be the ledger the local node started: either + // way its own proposal is obsolete (the ledger is closing, or catchup + // is about to trigger) + mCurrentValue.reset(); + mLedgerSeqProposing = 0; if (!mHerder.isTracking()) { @@ -1338,22 +1006,20 @@ HerderSCPDriver::logQuorumInformationAndUpdateMetrics(uint64_t index) } void -HerderSCPDriver::nominate(uint64_t slotIndex, NominationValueSupplier makeValue, - StellarValue const& previousValue, - bool constructValueNow) +HerderSCPDriver::startSlot(uint64_t slotIndex, ProposalValueSupplier makeValue, + bool constructValueNow) { ZoneScoped; - if (mLedgerSeqNominating != slotIndex) + if (mLedgerSeqProposing != slotIndex) { mCurrentValue.reset(); } - mLedgerSeqNominating = static_cast(slotIndex); + mLedgerSeqProposing = static_cast(slotIndex); if (constructValueNow && !mCurrentValue) { auto value = makeValue(); - // Publishing the tx set may externalize this slot, which stops its - // nomination. Do not restart it. - if (mLedgerSeqNominating != slotIndex) + // Publishing the tx set may externalize this slot. Do not restart it. + if (mLedgerSeqProposing != slotIndex) { return; } @@ -1361,28 +1027,85 @@ HerderSCPDriver::nominate(uint64_t slotIndex, NominationValueSupplier makeValue, // a leader. mCurrentValue = std::move(value); } - auto prevValue = xdr::xdr_to_opaque(previousValue); - mSCP.nominate( - slotIndex, - [this, slotIndex, - makeValue = std::move(makeValue)]() -> ValueWrapperPtr { - if (mLedgerSeqNominating != slotIndex) - { - return nullptr; - } - if (!mCurrentValue) - { - auto value = makeValue(); - // Publishing the tx set may externalize a pending value. - if (mLedgerSeqNominating != slotIndex) - { - return nullptr; - } - mCurrentValue = std::move(value); - } - return mCurrentValue; - }, - prevValue); + mSCP.startSlot(slotIndex, + [this, slotIndex, + makeValue = std::move(makeValue)]() -> ValueWrapperPtr { + if (mLedgerSeqProposing != slotIndex) + { + return nullptr; + } + if (!mCurrentValue) + { + auto value = makeValue(); + // Publishing the tx set may externalize a pending + // value. + if (mLedgerSeqProposing != slotIndex) + { + return nullptr; + } + mCurrentValue = std::move(value); + } + return mCurrentValue; + }); +} + +std::vector const& +HerderSCPDriver::getLeaderSchedule(uint64_t slotIndex) +{ + static std::vector const empty; + releaseAssert(threadIsMain()); + auto const& lcl = mLedgerManager.getLastClosedLedgerHeader(); + if (slotIndex != static_cast(lcl.header.ledgerSeq) + 1) + { + // The seed is the hash of the previous ledger's header: unknown for + // any slot but the next one. + return empty; + } + // Force a pending rebuild of the transitive quorum, if any + mPendingEnvelopes.isNodeDefinitelyInQuorum(mSCP.getLocalNodeID()); + auto const& quorum = mPendingEnvelopes.getCurrentlyTrackedQuorum(); + auto& cache = mLeaderSchedule; + if (cache.mSlotIndex != slotIndex || cache.mSeed != lcl.hash || + cache.mQuorumSize != quorum.size() || + cache.mQuorumRebuildCount != mPendingEnvelopes.getQuorumRebuildCount()) + { + std::vector nodes; + nodes.reserve(quorum.size()); + for (auto const& n : quorum) + { + nodes.emplace_back(n.first); + } + cache.mSchedule = computeLeaderSchedule(lcl.hash, std::move(nodes)); + cache.mSlotIndex = slotIndex; + cache.mSeed = lcl.hash; + cache.mQuorumSize = quorum.size(); + cache.mQuorumRebuildCount = mPendingEnvelopes.getQuorumRebuildCount(); + } + return cache.mSchedule; +} + +std::optional +HerderSCPDriver::getRoundLeader(uint64 slotIndex, uint32 round) +{ + auto const& schedule = getLeaderSchedule(slotIndex); + if (schedule.empty() || round == 0) + { + return std::nullopt; + } + return schedule[(round - 1) % schedule.size()]; +} + +bool +HerderSCPDriver::isProposalPending(uint64 slotIndex, NodeID const& leader) const +{ + return mPendingEnvelopes.hasPendingEnvelopeFrom(slotIndex, leader); +} + +bool +HerderSCPDriver::isLocalNodeRoundLeader(uint64_t slotIndex, uint32_t round) +{ + auto leader = getRoundLeader(slotIndex, round); + return leader && *leader == mSCP.getLocalNodeID(); } SCPQuorumSetPtr @@ -1432,22 +1155,44 @@ HerderSCPDriver::measureAndRecordBallotBlockedOnTxSet(uint64_t slotIndex, } void -HerderSCPDriver::nominatingValue(uint64_t slotIndex, Value const& value) +HerderSCPDriver::proposedValue(uint64_t slotIndex, uint32 round, + Value const& value) { - CLOG_DEBUG(Herder, "nominatingValue i:{} v: {}", slotIndex, + CLOG_DEBUG(Herder, "proposedValue i:{} round:{} v: {}", slotIndex, round, getValueString(value)); } void -HerderSCPDriver::updatedCandidateValue(uint64_t slotIndex, Value const& value) +HerderSCPDriver::enteredRound(uint64_t slotIndex, uint32 round, bool timedOut) { + auto it = mSCPExecutionTimes.find(slotIndex); + if (it != mSCPExecutionTimes.end()) + { + it->second.mMaxRound = std::max(it->second.mMaxRound, round); + } + if (round > 1) + { + // Only read the cached schedule here: this is called from inside SCP + // state changes (including state restore), where a quorum rebuild + // (database reads) does not belong. + auto const& cache = mLeaderSchedule; + std::string leader = "unknown"; + if (cache.mSlotIndex == slotIndex && !cache.mSchedule.empty()) + { + leader = toShortString( + cache.mSchedule[(round - 2) % cache.mSchedule.size()]); + } + CLOG_INFO(Herder, "Slot {} moved to round {}{} (round {} leader: {})", + slotIndex, round, timedOut ? " after a timeout" : "", + round - 1, leader); + } } void HerderSCPDriver::startedBallotProtocol(uint64_t slotIndex, SCPBallot const& ballot) { - recordSCPEvent(slotIndex, false); + recordBallotStart(slotIndex); // Prewarm the next ledger's signatures, after SCP has finished processing // (and emitting) this step: the mempool pull waits for the overlay. StellarValue sv; @@ -1560,40 +1305,34 @@ HerderSCPDriver::getExternalizeLag(NodeID const& id) const } void -HerderSCPDriver::recordNominationTrigger(uint64_t slotIndex) +HerderSCPDriver::recordTrigger(uint64_t slotIndex) { auto& timing = mSCPExecutionTimes[slotIndex]; - if (!timing.mTriggerStart && !timing.mNominationStart) + if (!timing.mTriggerStart) { timing.mTriggerStart = mApp.getClock().now(); + if (timing.mPrepareStart) + { + // The node already voted for the round leader's proposal before + // its own trigger fired. + mSCPMetrics.mTriggerToBallot.Update(std::chrono::nanoseconds(0)); + } } } void -HerderSCPDriver::recordSCPEvent(uint64_t slotIndex, bool isNomination) +HerderSCPDriver::recordBallotStart(uint64_t slotIndex) { - auto& timing = mSCPExecutionTimes[slotIndex]; VirtualClock::time_point start = mApp.getClock().now(); - if (isNomination) - { - if (!timing.mNominationStart) - { - timing.mNominationStart = start; - } - } - else + // The first entry into ballot ends the trigger-to-ballot interval + if (!timing.mPrepareStart && timing.mTriggerStart && + start >= *timing.mTriggerStart) { - // The first entry into ballot ends the trigger-to-ballot interval - if (!timing.mPrepareStart && timing.mTriggerStart && - start >= *timing.mTriggerStart) - { - mSCPMetrics.mTriggerToBallot.Update(start - *timing.mTriggerStart); - } - timing.mPrepareStart = - std::make_optional(start); + mSCPMetrics.mTriggerToBallot.Update(start - *timing.mTriggerStart); } + timing.mPrepareStart = std::make_optional(start); } void @@ -1686,33 +1425,20 @@ HerderSCPDriver::recordSCPExecutionMetrics(uint64_t slotIndex) auto& SCPTiming = SCPTimingIt->second; - mNominateTimeout.Update(SCPTiming.mNominationTimeoutCount); + mLeaderTimeout.Update(SCPTiming.mLeaderTimeoutCount); mPrepareTimeout.Update(SCPTiming.mPrepareTimeoutCount); - - if (SCPTiming.mNominationTimeoutCount > 0) + if (SCPTiming.mMaxRound > 0) { - auto const leaders = getSCP().getNominationLeaders(slotIndex); - std::string leaderStr; - for (auto const& leader : leaders) - { - if (!leaderStr.empty()) - { - leaderStr += ", "; - } - leaderStr += toShortString(leader); - } - CLOG_INFO(Herder, - "Nomination for slot {} timed out {} time(s) with " - "the following round leaders: [{}]", - slotIndex, SCPTiming.mNominationTimeoutCount, leaderStr); + mLeaderRound.Update(SCPTiming.mMaxRound); } - // Compute nomination time - if (SCPTiming.mNominationStart && SCPTiming.mPrepareStart) + if (SCPTiming.mLeaderTimeoutCount > 0) { - recordLogTiming(*SCPTiming.mNominationStart, *SCPTiming.mPrepareStart, - mSCPMetrics.mNominateToPrepare, "Nominate", threshold, - slotIndex); + CLOG_INFO(Herder, + "Slot {} timed out {} time(s) waiting for a round leader " + "(externalized in round {})", + slotIndex, SCPTiming.mLeaderTimeoutCount, + SCPTiming.mMaxRound); } // Compute prepare time @@ -2026,81 +1752,13 @@ HerderSCPDriver::TxSetValidityKeyHash::operator()( return res; } -uint64 -HerderSCPDriver::getNodeWeight(NodeID const& nodeID, SCPQuorumSet const& qset, - bool const isLocalNode) const -{ - releaseAssert(!mLedgerManager.isApplying()); - Config const& cfg = mApp.getConfig(); - bool const unsupportedProtocol = protocolVersionIsBefore( - mApp.getLedgerManager() - .getLastClosedLedgerHeader() - .header.ledgerVersion, - APPLICATION_SPECIFIC_NOMINATION_LEADER_ELECTION_PROTOCOL_VERSION); - if (unsupportedProtocol || !cfg.VALIDATOR_WEIGHT_CONFIG.has_value() || - cfg.FORCE_OLD_STYLE_LEADER_ELECTION) - { - // Fall back on old weight algorithm if any of the following are true: - // 1. The network has not yet upgraded to - // APPLICATION_SPECIFIC_NOMINATION_LEADER_ELECTION_PROTOCOL_VERSION, - // 2. The node is using manual quorum set configuration, or - // 3. The node has the FORCE_OLD_STYLE_LEADER_ELECTION flag - // set - return SCPDriver::getNodeWeight(nodeID, qset, isLocalNode); - } - - ValidatorWeightConfig const& vwc = - mApp.getConfig().VALIDATOR_WEIGHT_CONFIG.value(); - - auto entryIt = vwc.mValidatorEntries.find(nodeID); - if (entryIt == vwc.mValidatorEntries.end()) - { - // This shouldn't be possible as the validator entries should contain - // all validators in the config. For this to happen, `getNodeWeight` - // would have to be called with a non-validator `nodeID`. Throw if - // building tests, and otherwise fall back on the old algorithm. - throw std::runtime_error( - fmt::format(FMT_STRING("Validator entry not found for node {}"), - toShortString(nodeID))); - } - - ValidatorEntry const& entry = entryIt->second; - auto homeDomainSizeIt = vwc.mHomeDomainSizes.find(entry.mHomeDomain); - if (homeDomainSizeIt == vwc.mHomeDomainSizes.end()) - { - // This shouldn't be possible as the home domain sizes should contain - // all home domains in the config. For this to happen, `getNodeWeight` - // would have to be called with a non-validator, or the config parser - // would have to allow a validator without a home domain. Throw if - // building tests, and otherwise fall back on the old algorithm. - throw std::runtime_error( - fmt::format(FMT_STRING("Home domain size not found for domain {}"), - entry.mHomeDomain)); - } - - auto qualityWeightIt = vwc.mQualityWeights.find(entry.mQuality); - if (qualityWeightIt == vwc.mQualityWeights.end()) - { - // This shouldn't be possible as the quality weights should contain all - // quality levels in the config. - throw std::runtime_error( - fmt::format(FMT_STRING("Quality weight not found for quality {}"), - static_cast(entry.mQuality))); - } - - // Node's weight is its quality's weight divided by the number of nodes in - // its home domain - releaseAssert(homeDomainSizeIt->second > 0); - return qualityWeightIt->second / homeDomainSizeIt->second; -} - std::optional -HerderSCPDriver::getNominationTimeouts(uint64_t slotIndex) const +HerderSCPDriver::getLeaderTimeouts(uint64_t slotIndex) const { auto it = mSCPExecutionTimes.find(slotIndex); if (it != mSCPExecutionTimes.end()) { - return it->second.mNominationTimeoutCount; + return it->second.mLeaderTimeoutCount; } return std::nullopt; } diff --git a/src/herder/HerderSCPDriver.h b/src/herder/HerderSCPDriver.h index 047412be30..1067c0575f 100644 --- a/src/herder/HerderSCPDriver.h +++ b/src/herder/HerderSCPDriver.h @@ -34,11 +34,6 @@ class VirtualTimer; struct StellarValue; struct SCPEnvelope; -// First protocol version supporting the application-specific weight function -// for SCP leader election. -ProtocolVersion constexpr APPLICATION_SPECIFIC_NOMINATION_LEADER_ELECTION_PROTOCOL_VERSION = - ProtocolVersion::V_22; - class HerderSCPDriver : public SCPDriver { public: @@ -57,8 +52,10 @@ class HerderSCPDriver : public SCPDriver } void recordSCPExecutionMetrics(uint64_t slotIndex); - void recordNominationTrigger(uint64_t slotIndex); - void recordSCPEvent(uint64_t slotIndex, bool isNomination); + // The local trigger for slotIndex fired. + void recordTrigger(uint64_t slotIndex); + // The local node entered the ballot protocol for slotIndex. + void recordBallotStart(uint64_t slotIndex); void recordSCPExternalizeEvent(uint64_t slotIndex, NodeID const& id, bool forceUpdateSelf); @@ -72,9 +69,7 @@ class HerderSCPDriver : public SCPDriver // value validation SCPDriver::ValidationLevel validateValue(uint64_t slotIndex, Value const& value, - bool nomination) const override; - ValueWrapperPtr extractValidValue(uint64_t slotIndex, - Value const& value) override; + bool proposal) const override; // value marshaling std::string toShortString(NodeID const& pk) const override; @@ -106,45 +101,47 @@ class HerderSCPDriver : public SCPDriver std::function cb) override; void stopTimer(uint64 slotIndex, int timerID) override; - std::chrono::milliseconds computeTimeout(uint32 roundNumber, - bool isNomination) override; -#ifdef BUILD_TESTS - std::chrono::milliseconds getNominationEmitDelayForTesting() const override; -#endif + std::chrono::milliseconds computeTimeout(uint32 roundNumber) override; // hashing support Hash getHashOf(std::vector> const& vals) const override; // core SCP - ValueWrapperPtr - combineCandidates(uint64_t slotIndex, - ValueWrapperPtrSet const& candidates) override; void valueExternalized(uint64_t slotIndex, Value const& value) override; - bool hasUpgrades(Value const& v) override; - ValueWrapperPtr stripAllUpgrades(Value const& v) override; - uint32_t getUpgradeNominationTimeoutLimit() const override; + // Deterministic leader schedule: the leader of `round` for `slotIndex`. + // The schedule is a permutation of the transitive quorum seeded by the + // hash of the last closed ledger header, so it is only defined for the + // next ledger to close (LCL+1). + bool isProposalPending(uint64 slotIndex, + NodeID const& leader) const override; + std::optional getRoundLeader(uint64 slotIndex, + uint32 round) override; + // The whole schedule for slotIndex (round r is led by entry + // (r - 1) % size). Empty when the schedule is not defined for the slot. + std::vector const& getLeaderSchedule(uint64_t slotIndex); + // True if the local node leads `round` of slotIndex. + bool isLocalNodeRoundLeader(uint64_t slotIndex, uint32_t round); // Update metrics indicating that a value was replaced with an empty-tx-set // value for slotIndex. void noteEmptyTxSetValueReplaced(uint64_t slotIndex) override; - // Start nomination for slotIndex. SCP calls makeValue only when the local - // node is a round leader that needs its own value, unless - // constructValueNow is set: then the value is constructed before - // nomination starts (and reused if this node becomes a leader). - // previousValue is the value from slotIndex-1 - void nominate(uint64_t slotIndex, NominationValueSupplier makeValue, - StellarValue const& previousValue, - bool constructValueNow = false); + // Start consensus on slotIndex. SCP calls makeValue only when the local + // node leads a round and needs its own value, unless constructValueNow + // is set: then the value is constructed first (and reused if this node + // becomes a leader). + void startSlot(uint64_t slotIndex, ProposalValueSupplier makeValue, + bool constructValueNow = false); SCPQuorumSetPtr getQSet(Hash const& qSetHash) override; // listeners void ballotDidHearFromQuorum(uint64_t slotIndex, SCPBallot const& ballot) override; - void nominatingValue(uint64_t slotIndex, Value const& value) override; - void updatedCandidateValue(uint64_t slotIndex, Value const& value) override; + void proposedValue(uint64_t slotIndex, uint32 round, + Value const& value) override; + void enteredRound(uint64_t slotIndex, uint32 round, bool timedOut) override; void startedBallotProtocol(uint64_t slotIndex, SCPBallot const& ballot) override; void acceptedBallotPrepared(uint64_t slotIndex, @@ -199,13 +196,6 @@ class HerderSCPDriver : public SCPDriver getTxSetDownloadWaitTime(Value const& v) const override; std::chrono::milliseconds getTxSetDownloadTimeout() const override; - // Application-specific weight function. This function uses the quality - // levels from automatic quorum set generation to determine the weight of a - // validator. It is designed to ensure that: - // 1. Orgs of equal quality have equal chances of winning leader election. - // 2. Higher quality orgs win more frequently than lower quality orgs. - uint64 getNodeWeight(NodeID const& nodeID, SCPQuorumSet const& qset, - bool isLocalNode) const override; // For caching TxSet validity. Consist of {lcl.hash, txSetHash, // lowerBoundCloseTimeOffset, upperBoundCloseTimeOffset} using TxSetValidityKey = std::tuple; @@ -219,12 +209,14 @@ class HerderSCPDriver : public SCPDriver LedgerHeaderHistoryEntry const& lcl, ApplyTimeOffset closeTimeOffset) const; - // Get the number of nomination timeouts that occurred for a given slot - std::optional getNominationTimeouts(uint64_t slotIndex) const; + // Get the number of times the local node timed out waiting for a round + // leader in a given slot + std::optional getLeaderTimeouts(uint64_t slotIndex) const; // Elapsed local time from the trigger through entry into ballot, measured - // on the steady clock. Includes construction and incomplete nomination - // rounds. Missing timing history does not establish any allowance. + // on the steady clock. Includes construction and rounds whose leader + // never showed up. Missing timing history does not establish any + // allowance. std::chrono::milliseconds getTriggerToBallotDuration(uint64_t slotIndex) const; @@ -267,15 +259,12 @@ class HerderSCPDriver : public SCPDriver medida::Meter& mValueValid; medida::Meter& mValueInvalid; - // listeners - medida::Meter& mCombinedCandidates; - - // Timers for nomination and ballot protocols - medida::Timer& mNominateToPrepare; + // Timer for the ballot protocol medida::Timer& mPrepareToExternalize; - // Local trigger to first entry into ballot (proposal construction - // plus nomination); the drift allowance of the trigger anchor + // Local trigger to first entry into ballot (proposal construction, or + // waiting for the round leader's proposal); the drift allowance of + // the trigger anchor medida::Timer& mTriggerToBallot; // Timers tracking externalize messages @@ -300,8 +289,10 @@ class HerderSCPDriver : public SCPDriver SCPMetrics mSCPMetrics; - // Nomination timeouts per ledger - medida::Histogram& mNominateTimeout; + // Timeouts waiting for a round leader, per ledger + medida::Histogram& mLeaderTimeout; + // Round in which the ledger externalized (1 = first leader) + medida::Histogram& mLeaderRound; // Prepare timeouts per ledger medida::Histogram& mPrepareTimeout; // Unique values referenced per ledger @@ -313,11 +304,12 @@ class HerderSCPDriver : public SCPDriver struct SCPTiming { std::optional mTriggerStart; - std::optional mNominationStart; std::optional mPrepareStart; - // Nomination timeouts before first prepare - int64_t mNominationTimeoutCount{0}; + // Timeouts waiting for a round leader's proposal + int64_t mLeaderTimeoutCount{0}; + // Highest round the local node entered + uint32_t mMaxRound{0}; // Prepare timeouts before externalize int64_t mPrepareTimeoutCount{0}; @@ -331,13 +323,25 @@ class HerderSCPDriver : public SCPDriver }; // Map of time points for each slot to measure key protocol metrics: - // * nomination to first prepare + // * trigger to first prepare // * first prepare to externalize std::map mSCPExecutionTimes; - uint32_t mLedgerSeqNominating; + // Slot the local node started, and the value it built for it (if any) + uint32_t mLedgerSeqProposing; ValueWrapperPtr mCurrentValue; + // Leader schedule of the next ledger to close + struct LeaderScheduleCache + { + uint64_t mSlotIndex{0}; + Hash mSeed; + size_t mQuorumSize{0}; + uint64_t mQuorumRebuildCount{0}; + std::vector mSchedule; + }; + LeaderScheduleCache mLeaderSchedule; + // timers used by SCP // indexed by slotIndex, timerID std::map>> mSCPTimers; @@ -354,7 +358,7 @@ class HerderSCPDriver : public SCPDriver SCPDriver::ValidationLevel validateValueAgainstLocalState(uint64_t slotIndex, StellarValue const& sv, - bool nomination) const; + bool proposal) const; SCPDriver::ValidationLevel validatePastOrFutureValue(uint64_t slotIndex, StellarValue const& b, @@ -379,6 +383,6 @@ class HerderSCPDriver : public SCPDriver bool deserializeAndValidateStellarValue(uint64_t slotIndex, Value const& value, StellarValue& sv) const; - void extractValidUpgrades(StellarValue& sv, bool nomination) const; + void extractValidUpgrades(StellarValue& sv, bool proposal) const; }; } diff --git a/src/herder/LeaderSchedule.cpp b/src/herder/LeaderSchedule.cpp new file mode 100644 index 0000000000..cc9599a718 --- /dev/null +++ b/src/herder/LeaderSchedule.cpp @@ -0,0 +1,38 @@ +// Copyright 2026 Stellar Development Foundation and contributors. Licensed +// under the Apache License, Version 2.0. See the COPYING file at the root +// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 + +#include "herder/LeaderSchedule.h" +#include "crypto/SHA.h" +#include "util/XDROperators.h" +#include "xdrpp/marshal.h" +#include +#include + +namespace stellar +{ + +std::vector +computeLeaderSchedule(Hash const& seed, std::vector nodes) +{ + std::vector> keyed; + keyed.reserve(nodes.size()); + for (auto& node : nodes) + { + SHA256 hasher; + hasher.add(seed); + hasher.add(xdr::xdr_to_opaque(node)); + keyed.emplace_back(hasher.finish(), std::move(node)); + } + std::sort(keyed.begin(), keyed.end()); + keyed.erase(std::unique(keyed.begin(), keyed.end()), keyed.end()); + + std::vector res; + res.reserve(keyed.size()); + for (auto& k : keyed) + { + res.emplace_back(std::move(k.second)); + } + return res; +} +} diff --git a/src/herder/LeaderSchedule.h b/src/herder/LeaderSchedule.h new file mode 100644 index 0000000000..9042d90eaa --- /dev/null +++ b/src/herder/LeaderSchedule.h @@ -0,0 +1,26 @@ +// Copyright 2026 Stellar Development Foundation and contributors. Licensed +// under the Apache License, Version 2.0. See the COPYING file at the root +// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 + +#pragma once + +#include "xdr/Stellar-types.h" +#include + +namespace stellar +{ + +// Deterministic leader schedule (experimental; replaces SCP nomination). +// +// Returns `nodes` (duplicates removed) ordered by SHA256(seed || nodeID), +// ties (impossible in practice) broken by node ID. Round r (1-based) of the +// slot is led by entry (r - 1) % size, so every node gets a turn before any +// node leads twice. +// +// The order depends only on `seed` and the *set* of nodes, so every node that +// agrees on both computes the same schedule: with a symmetric quorum, seeding +// with the hash of the last closed ledger header gives the whole network the +// same leader for each round of the next ledger. +std::vector computeLeaderSchedule(Hash const& seed, + std::vector nodes); +} diff --git a/src/herder/PendingEnvelopes.cpp b/src/herder/PendingEnvelopes.cpp index 6c3a3b5af5..82b35f300f 100644 --- a/src/herder/PendingEnvelopes.cpp +++ b/src/herder/PendingEnvelopes.cpp @@ -1164,6 +1164,7 @@ PendingEnvelopes::rebuildQuorumTrackerState() { // rebuild quorum information using data sources starting with the // freshest source + ++mQuorumRebuildCount; mQuorumTracker.rebuild([&](NodeID const& id) -> SCPQuorumSetPtr { SCPQuorumSetPtr res; if (id == mHerder.getSCP().getLocalNodeID()) @@ -1195,6 +1196,32 @@ PendingEnvelopes::rebuildQuorumTrackerState() }); } +bool +PendingEnvelopes::hasPendingEnvelopeFrom(uint64 slotIndex, + NodeID const& node) const +{ + auto it = mEnvelopes.find(slotIndex); + if (it == mEnvelopes.end()) + { + return false; + } + for (auto const& f : it->second.mFetchingEnvelopes) + { + if (f.first.statement.nodeID == node) + { + return true; + } + } + for (auto const& r : it->second.mReadyEnvelopes) + { + if (r->getStatement().nodeID == node) + { + return true; + } + } + return false; +} + QuorumTracker::QuorumMap const& PendingEnvelopes::getCurrentlyTrackedQuorum() const { diff --git a/src/herder/PendingEnvelopes.h b/src/herder/PendingEnvelopes.h index 75f5973085..0011b5cda1 100644 --- a/src/herder/PendingEnvelopes.h +++ b/src/herder/PendingEnvelopes.h @@ -94,6 +94,7 @@ class PendingEnvelopes RandomEvictionCache mValueSizeCache; bool mRebuildQuorum; + uint64_t mQuorumRebuildCount{0}; QuorumTracker mQuorumTracker; medida::Counter& mProcessedCount; @@ -256,6 +257,17 @@ class PendingEnvelopes bool isNodeDefinitelyInQuorum(NodeID const& node); void rebuildQuorumTrackerState(); + // True if an envelope of `node` for `slotIndex` is being fetched (waiting + // for its tx set or quorum set) or is ready but not yet processed by SCP. + bool hasPendingEnvelopeFrom(uint64 slotIndex, NodeID const& node) const; + // Incremented every time the transitive quorum is rebuilt from scratch. + // Together with the quorum's size (it only grows between rebuilds) this + // identifies its node set. + uint64_t + getQuorumRebuildCount() const + { + return mQuorumRebuildCount; + } QuorumTracker::QuorumMap const& getCurrentlyTrackedQuorum() const; // updates internal state when an envelope was successfully processed diff --git a/src/herder/test/HerderTests.cpp b/src/herder/test/HerderTests.cpp index 0fc00e6b43..f151cbe81c 100644 --- a/src/herder/test/HerderTests.cpp +++ b/src/herder/test/HerderTests.cpp @@ -2675,23 +2675,15 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) return makeTxUpgradePair(herder, txSet, closeTime, emptyUpgradeSteps); }; auto makeEnvelope = [&s](HerderImpl& herder, TxPair const& p, Hash qSetHash, - uint64_t slotIndex, bool nomination) { + uint64_t slotIndex) { // herder must want the TxSet before receiving it, so we are sending it // fake envelope auto envelope = SCPEnvelope{}; envelope.statement.slotIndex = slotIndex; - if (nomination) - { - envelope.statement.pledges.type(SCP_ST_NOMINATE); - envelope.statement.pledges.nominate().votes.push_back(p.first); - envelope.statement.pledges.nominate().quorumSetHash = qSetHash; - } - else - { - envelope.statement.pledges.type(SCP_ST_PREPARE); - envelope.statement.pledges.prepare().ballot.value = p.first; - envelope.statement.pledges.prepare().quorumSetHash = qSetHash; - } + envelope.statement.pledges.type(SCP_ST_PREPARE); + envelope.statement.pledges.prepare().ballot.counter = 1; + envelope.statement.pledges.prepare().ballot.value = p.first; + envelope.statement.pledges.prepare().quorumSetHash = qSetHash; envelope.statement.nodeID = s.getPublicKey(); herder.signEnvelope(s, envelope); return envelope; @@ -2708,156 +2700,6 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) return makeTxSetFromTransactions(txs, *app, ApplyTimeOffset{}); }; - SECTION("combineCandidates") - { - auto& herder = static_cast(app->getHerder()); - - ValueWrapperPtrSet candidates; - - auto addToCandidates = [&](TxPair const& p) { - auto envelope = makeEnvelope( - herder, p, {}, herder.trackingConsensusLedgerIndex() + 1, true); - REQUIRE(herder.recvSCPEnvelope(envelope) == - Herder::ENVELOPE_STATUS_FETCHING); - REQUIRE(herder.recvTxSet(p.second->getContentsHash(), p.second)); - auto v = herder.getHerderSCPDriver().wrapValue(p.first); - candidates.emplace(v); - }; - - struct CandidateSpec - { - int const n; - int const nbOps; - uint32 const feeMulti; - TimePoint const closeTime; - std::optional const baseFeeIncrement; - }; - - std::vector txSetHashes; - std::vector txSetSizes; - std::vector txSetOpSizes; - std::vector closeTimes; - std::vector baseFees; - - auto addCandidateThenTest = [&](CandidateSpec const& spec) { - // Create a transaction set using the given parameters, combine - // it with the given closeTime and optionally a given base fee - // increment, and make it into a StellarValue to add to the list - // of candidates so far. Keep track of the hashes and sizes and - // operation sizes of all the transaction sets, all of the close - // times, and all of the base fee upgrades that we've seen, so that - // we can compute the expected result of combining all the - // candidates so far. (We're using base fees simply as one example - // of a type of upgrade, whose expected result is the maximum of all - // candidates'.) - auto [txSet, applicableTxSet] = - makeTransactions(spec.n, spec.nbOps, spec.feeMulti); - txSetHashes.push_back(txSet->getContentsHash()); - txSetSizes.push_back(applicableTxSet->size(lcl.header)); - txSetOpSizes.push_back(applicableTxSet->sizeOpTotal()); - closeTimes.push_back(spec.closeTime); - if (spec.baseFeeIncrement) - { - auto const baseFee = - lcl.header.baseFee + *spec.baseFeeIncrement; - baseFees.push_back(baseFee); - LedgerUpgrade ledgerUpgrade; - ledgerUpgrade.type(LEDGER_UPGRADE_BASE_FEE); - ledgerUpgrade.newBaseFee() = baseFee; - Value upgrade(xdr::xdr_to_opaque(ledgerUpgrade)); - SVUpgrades upgrades; - upgrades.emplace_back(upgrade.begin(), upgrade.end()); - addToCandidates( - makeTxUpgradePair(herder, txSet, spec.closeTime, upgrades)); - } - else - { - addToCandidates(makeTxPair(herder, txSet, spec.closeTime)); - } - - // Compute the expected transaction set, close time, and upgrade - // vector resulting from combining all the candidates so far. - auto const bestTxSetIndex = std::distance( - txSetSizes.begin(), - std::max_element(txSetSizes.begin(), txSetSizes.end())); - REQUIRE(txSetSizes.size() == closeTimes.size()); - auto const expectedHash = txSetHashes[bestTxSetIndex]; - auto const expectedCloseTime = closeTimes[bestTxSetIndex]; - SVUpgrades expectedUpgradeVector; - if (!baseFees.empty()) - { - LedgerUpgrade expectedLedgerUpgrade; - expectedLedgerUpgrade.type(LEDGER_UPGRADE_BASE_FEE); - expectedLedgerUpgrade.newBaseFee() = - *std::max_element(baseFees.begin(), baseFees.end()); - Value const expectedUpgradeValue( - xdr::xdr_to_opaque(expectedLedgerUpgrade)); - expectedUpgradeVector.emplace_back(expectedUpgradeValue.begin(), - expectedUpgradeValue.end()); - } - - // Combine all the candidates seen so far, and extract the - // returned StellarValue. - ValueWrapperPtr v = - herder.getHerderSCPDriver().combineCandidates(1, candidates); - StellarValue sv; - xdr::xdr_from_opaque(v->getValue(), sv); - - // Compare the returned StellarValue's contents with the - // expected ones that we computed above. - REQUIRE(isSignedStellarValue(sv)); - REQUIRE(sv.txSetHash == expectedHash); - REQUIRE(sv.closeTime == expectedCloseTime); - REQUIRE(sv.upgrades == expectedUpgradeVector); - }; - - // Test some list of candidates, comparing the output of - // combineCandidates() and the one we compute at each step. - - std::vector const specs{ - {0, 1, 100, 10, std::nullopt}, - {10, 1, 100, 5, std::make_optional(1)}, - {5, 3, 100, 20, std::make_optional(2)}, - {7, 2, 5, 30, std::make_optional(3)}}; - - std::for_each(specs.begin(), specs.end(), addCandidateThenTest); - - auto const bestTxSetIndex = std::distance( - txSetSizes.begin(), - std::max_element(txSetSizes.begin(), txSetSizes.end())); - REQUIRE(txSetOpSizes[bestTxSetIndex] == expectedOps); - - auto txSetL = makeTransactions(maxTxSetSize, 1, 101).first; - addToCandidates(makeTxPair(herder, txSetL, 20)); - auto txSetL2 = makeTransactions(maxTxSetSize, 1, 1000).first; - addToCandidates(makeTxPair(herder, txSetL2, 20)); - auto v = herder.getHerderSCPDriver().combineCandidates(1, candidates); - StellarValue sv; - xdr::xdr_from_opaque(v->getValue(), sv); - REQUIRE(isSignedStellarValue(sv)); - REQUIRE(sv.txSetHash == txSetL2->getContentsHash()); - - if (protocolVersionStartsFrom(protocolVersion, - MS_CLOSE_TIME_PROTOCOL_VERSION)) - { - // Test that the winning candidate keeps its full ms close time - // along with the redundant whole-second close time. - auto txSetL3 = makeTransactions(maxTxSetSize, 1, 2000).first; - StellarValue msVal = herder.makeStellarValue( - txSetL3->getContentsHash(), makeConsensusTime(20, 7), - emptyUpgradeSteps, root->getSecretKey()); - addToCandidates(TxPair{xdr::xdr_to_opaque(msVal), txSetL3}); - auto v2 = - herder.getHerderSCPDriver().combineCandidates(1, candidates); - StellarValue sv2; - xdr::xdr_from_opaque(v2->getValue(), sv2); - REQUIRE(sv2.ext.v() == STELLAR_VALUE_SIGNED_MS); - REQUIRE(sv2.closeTime == 20); - REQUIRE(getConsensusTime(sv2).milliseconds() == 20007); - REQUIRE(sv2.txSetHash == txSetL3->getContentsHash()); - } - } - SECTION("validateValue signatures") { auto& herder = static_cast(app->getHerder()); @@ -2869,7 +2711,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) { // make sure that txSet0 is loaded auto p = makeTxPair(herder, txSet0, ct); - auto envelope = makeEnvelope(herder, p, {}, seq, true); + auto envelope = makeEnvelope(herder, p, {}, seq); REQUIRE(herder.recvSCPEnvelope(envelope) == Herder::ENVELOPE_STATUS_FETCHING); REQUIRE(herder.recvTxSet(txSet0->getContentsHash(), txSet0)); @@ -2887,19 +2729,19 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) } SECTION("invalid") { - auto checkInvalid = [&](StellarValue const& sv, bool nomination) { + auto checkInvalid = [&](StellarValue const& sv, bool proposal) { auto v = xdr::xdr_to_opaque(sv); - REQUIRE(scp.validateValue(seq, v, nomination) == + REQUIRE(scp.validateValue(seq, v, proposal) == SCPDriver::kInvalidValue); }; - auto testInvalidValue = [&](bool isNomination) { + auto testInvalidValue = [&](bool isProposal) { SECTION("basic value") { auto basicVal = StellarValue(txSet0->getContentsHash(), ct, emptyUpgradeSteps, STELLAR_VALUE_BASIC); - checkInvalid(basicVal, isNomination); + checkInvalid(basicVal, isProposal); } SECTION("signed value") { @@ -2911,28 +2753,28 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) SECTION("missing signature") { getLcValueSignature(sv).signature.clear(); - checkInvalid(sv, isNomination); + checkInvalid(sv, isProposal); } SECTION("wrong signature") { getLcValueSignature(sv).signature[0] ^= 1; - checkInvalid(sv, isNomination); + checkInvalid(sv, isProposal); } SECTION("wrong signature 2") { getLcValueSignature(sv).nodeID.ed25519()[0] ^= 1; - checkInvalid(sv, isNomination); + checkInvalid(sv, isProposal); } } }; - SECTION("nomination") + SECTION("proposal") { - testInvalidValue(/* isNomination */ true); + testInvalidValue(/* isProposal */ true); } SECTION("ballot") { - testInvalidValue(/* isNomination */ false); + testInvalidValue(/* isProposal */ false); } } @@ -2945,10 +2787,6 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) SCPDriver::kInvalidValue); REQUIRE(scp.validateValue(seq, v, false) == SCPDriver::kInvalidValue); - - ValueWrapperPtr extracted; - REQUIRE_NOTHROW(extracted = scp.extractValidValue(seq, v)); - REQUIRE(extracted == nullptr); }; SECTION("signed value with empty-tx-set hash") @@ -2985,13 +2823,13 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) // once the protocol allows them. This is the assertion that // catches an inverted check in deserializeAndValidateStellarValue. REQUIRE(scp.validateValue(seq, emptyTxSetValue, - /*nomination=*/false) == + /*proposal=*/false) == (allowed ? SCPDriver::kFullyValidatedValue : SCPDriver::kInvalidValue)); - // Nomination path: empty-tx-set values are rejected by design. + // Proposal path: empty-tx-set values are rejected by design. REQUIRE(scp.validateValue(seq, emptyTxSetValue, - /*nomination=*/true) == + /*proposal=*/true) == SCPDriver::kInvalidValue); } } @@ -3048,7 +2886,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) emptyUpgradeSteps, root->getSecretKey()); auto val = TxPair{xdr::xdr_to_opaque(sv), txSet}; auto const seq = herder.trackingConsensusLedgerIndex() + 1; - auto envelope = makeEnvelope(herder, val, {}, seq, true); + auto envelope = makeEnvelope(herder, val, {}, seq); REQUIRE(herder.recvSCPEnvelope(envelope) == Herder::ENVELOPE_STATUS_FETCHING); REQUIRE(herder.recvTxSet(txSet->getContentsHash(), txSet)); @@ -3122,7 +2960,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) // All hits during the whole SCP round. One of them is the validity // check that determines whether or not to replace the transaction set // with an empty one (CAP-0083). - uint64_t const expectedHits = 11; + uint64_t const expectedHits = 9; REQUIRE(cache.getCounters().mHits == expectedHits); // One miss from the initial makeTxSetFromTransactions REQUIRE(cache.getCounters().mMisses == 1); @@ -3173,13 +3011,10 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) auto p2 = makeTxPair(herder, transactions1, 10); // use current + 1 to allow for any value (old values get filtered more) auto lseq = herder.trackingConsensusLedgerIndex() + 1; - auto saneEnvelopeQ1T1 = - makeEnvelope(herder, p1, saneQSet1Hash, lseq, true); - auto saneEnvelopeQ1T2 = - makeEnvelope(herder, p2, saneQSet1Hash, lseq, true); - auto saneEnvelopeQ2T1 = - makeEnvelope(herder, p1, saneQSet2Hash, lseq, true); - auto bigEnvelope = makeEnvelope(herder, p1, bigQSetHash, lseq, true); + auto saneEnvelopeQ1T1 = makeEnvelope(herder, p1, saneQSet1Hash, lseq); + auto saneEnvelopeQ1T2 = makeEnvelope(herder, p2, saneQSet1Hash, lseq); + auto saneEnvelopeQ2T1 = makeEnvelope(herder, p1, saneQSet2Hash, lseq); + auto bigEnvelope = makeEnvelope(herder, p1, bigQSetHash, lseq); TxSetXDRFrameConstPtr malformedTxSet; if (transactions1->isGeneralizedTxSet()) @@ -3204,7 +3039,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) } auto malformedTxSetPair = makeTxPair(herder, malformedTxSet, 10); auto malformedTxSetEnvelope = - makeEnvelope(herder, malformedTxSetPair, saneQSet1Hash, lseq, true); + makeEnvelope(herder, malformedTxSetPair, saneQSet1Hash, lseq); SECTION("return FETCHING until fetched") { @@ -3357,7 +3192,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) { // make sure that txSet0 is loaded auto p = makeTxPair(herder, txSet0, ct); - auto envelope = makeEnvelope(herder, p, {}, seq, true); + auto envelope = makeEnvelope(herder, p, {}, seq); REQUIRE(herder.recvSCPEnvelope(envelope) == Herder::ENVELOPE_STATUS_FETCHING); REQUIRE(herder.recvTxSet(txSet0->getContentsHash(), txSet0)); @@ -3365,7 +3200,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) auto check = [&](StellarValue const& sv, bool expectValid) { auto v = xdr::xdr_to_opaque(sv); - auto res = scp.validateValue(seq, v, /*nomination=*/true); + auto res = scp.validateValue(seq, v, /*proposal=*/true); if (expectValid) { REQUIRE(res == SCPDriver::kFullyValidatedValue); @@ -3428,14 +3263,14 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) REQUIRE(scp.validateValue( pastSlot, xdr::xdr_to_opaque(makeLegacySignedValue(ct)), - /*nomination=*/false) == + /*proposal=*/false) == SCPDriver::kMaybeValidNotCurrentValue); // Newer-than-externalized close time for an older slot: // rejected REQUIRE(scp.validateValue( pastSlot, xdr::xdr_to_opaque(makeLegacySignedValue(ct + 60)), - /*nomination=*/false) == SCPDriver::kInvalidValue); + /*proposal=*/false) == SCPDriver::kInvalidValue); } SECTION("legacy signed value rejected for future slots") { @@ -3445,7 +3280,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) REQUIRE(scp.validateValue( seq + 1, xdr::xdr_to_opaque(makeLegacySignedValue(ct)), - /*nomination=*/false) == SCPDriver::kInvalidValue); + /*proposal=*/false) == SCPDriver::kInvalidValue); } SECTION("legacy empty-tx-set value rejected for ratifiable slots") { @@ -3455,7 +3290,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) xdr::xdr_from_opaque(legacyEmpty, esv); REQUIRE(esv.ext.v() == STELLAR_VALUE_EMPTY_TX_SET); REQUIRE(scp.validateValue(seq, legacyEmpty, - /*nomination=*/false) == + /*proposal=*/false) == SCPDriver::kInvalidValue); } SECTION("closeTime inconsistent with closeTimeMs") @@ -3576,7 +3411,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) rand_uniform(1, 999)), emptyUpgradeSteps, s); auto val = TxPair{xdr::xdr_to_opaque(msv), txSet}; - auto envelope = makeEnvelope(herder, val, {}, zSeq, true); + auto envelope = makeEnvelope(herder, val, {}, zSeq); REQUIRE(herder.recvSCPEnvelope(envelope) == Herder::ENVELOPE_STATUS_FETCHING); REQUIRE(herder.recvTxSet(txSet->getContentsHash(), txSet)); @@ -3673,7 +3508,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) // triggers catchup herder.lostSync(); REQUIRE(scp.validateValue(seq + 1, xdr::xdr_to_opaque(msv), - /*nomination=*/false) == + /*proposal=*/false) == SCPDriver::kMaybeValidNotCurrentValue); } } @@ -3691,64 +3526,6 @@ TEST_CASE("SCP Driver", "[herder][acceptance]") } } -// Test combineCandidates handling of candidates where -// previousLedgerHash != LCL.hash -TEST_CASE("combineCandidates with mismatched previousLedgerHash candidate", - "[herder][bug]") -{ - Config cfg(getTestConfig()); - - VirtualClock clock; - auto app = createTestApplication(clock, cfg); - auto& herder = dynamic_cast(app->getHerder()); - auto& pe = herder.getPendingEnvelopes(); - auto& driver = herder.getHerderSCPDriver(); - - auto const& lcl = app->getLedgerManager().getLastClosedLedgerHeader(); - uint32_t const ver = lcl.header.ledgerVersion; - TimePoint const closeTime = lcl.header.scpValue.closeTime + 1; - uint64_t const slotIndex = lcl.header.ledgerSeq + 1; - - // Two structurally-valid empty tx sets that differ only in - // previousLedgerHash. - auto goodTxSet = TxSetXDRFrame::makeEmpty(lcl.hash, ver); // matches LCL - auto badTxSet = TxSetXDRFrame::makeEmpty(sha256("not the LCL hash"), - ver); // mismatched - - ValueWrapperPtrSet candidates; - // Register the tx set so combineCandidates' getTxSet() returns it, then add - // a candidate value referencing it. - auto addCandidate = [&](TxSetXDRFrameConstPtr const& txSet) { - pe.addTxSet(txSet->getContentsHash(), slotIndex, txSet); - StellarValue sv = herder.makeStellarValue( - txSet->getContentsHash(), makeConsensusTime(closeTime), - emptyUpgradeSteps, cfg.NODE_SEED); - candidates.emplace(driver.wrapValue(xdr::xdr_to_opaque(sv))); - }; - auto combinedTxSetHash = [&]() { - ValueWrapperPtr result = - driver.combineCandidates(slotIndex, candidates); - StellarValue sv; - xdr::xdr_from_opaque(result->getValue(), sv); - return sv.txSetHash; - }; - - SECTION("prefer applicable candidate over mismatched candidate") - { - addCandidate(goodTxSet); - addCandidate(badTxSet); - REQUIRE(combinedTxSetHash() == goodTxSet->getContentsHash()); - } - - SECTION("all candidates have mismatched previousLedgerHash") - { - // If the *only* option is a candidate with a mismatched - // previousLedgerHash, choose it. - addCandidate(badTxSet); - REQUIRE(combinedTxSetHash() == badTxSet->getContentsHash()); - } -} - TEST_CASE("SCP checkpoint", "[catchup][herder]") { auto networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); @@ -4728,11 +4505,14 @@ TEST_CASE("SCP message capture from previous ledger", "[herder]") auto networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); // A and B keep closing ledgers in real time while C is fed slot 2 by // hand. Use a longer close time so that C's slot-2 EXTERNALIZE reaches - // A and B before they close ledger 3. + // A and B before they close ledger 3. C is not connected, so it only has + // slot 2's (empty) transaction set once it built one itself, i.e. once it + // led a round: with three nodes that can take until round 3 (two leader + // timeouts, 3 s). auto confGen = [](int i) { auto cfg = getTestConfig(i); cfg.ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING = true; - cfg.ARTIFICIALLY_SET_CLOSE_TIME_FOR_TESTING = 5000; + cfg.ARTIFICIALLY_SET_CLOSE_TIME_FOR_TESTING = 10000; return cfg; }; auto simulation = std::make_shared(networkID, confGen); @@ -4796,6 +4576,19 @@ TEST_CASE("SCP message capture from previous ledger", "[herder]") ++actualTypes[env.statement.pledges.type()]; } + std::string actual; + for (auto const& [type, count] : actualTypes) + { + actual += + fmt::format("type {}: {}; ", static_cast(type), count); + } + UNSCOPED_INFO("scphistory of ledger " + << ledgerNum << " on " + << node->getConfig().toShortString( + node->getConfig().NODE_SEED.getPublicKey()) + << ": " << actual << "(LCL " + << node->getLedgerManager().getLastClosedLedgerNum() + << ")"); return actualTypes == expectedTypes; }; @@ -4829,13 +4622,29 @@ TEST_CASE("SCP message capture from previous ledger", "[herder]") std::vector BEnvs = herderB.getSCP().getLatestMessagesSend(2); // Pass A and B's messages to C + auto const tFeedC = A->getClock().now(); + UNSCOPED_INFO( + "feeding C: A LCL " + << A->getLedgerManager().getLastClosedLedgerNum() << ", C round " + << dynamic_cast(C->getHerder()).getSCP().getLeaderRound(2) + << ", C sent " + << dynamic_cast(C->getHerder()) + .getSCP() + .getLatestMessagesSend(2) + .size()); for (auto const& env : AEnvs) { - C->getHerder().recvSCPEnvelope(env); + auto st = C->getHerder().recvSCPEnvelope(env); + UNSCOPED_INFO("C status for A's message type " + << static_cast(env.statement.pledges.type()) << ": " + << static_cast(st)); } for (auto const& env : BEnvs) { - C->getHerder().recvSCPEnvelope(env); + auto st = C->getHerder().recvSCPEnvelope(env); + UNSCOPED_INFO("C status for B's message type " + << static_cast(env.statement.pledges.type()) << ": " + << static_cast(st)); } // Crank C until it is on ledger 2 @@ -4843,15 +4652,31 @@ TEST_CASE("SCP message capture from previous ledger", "[herder]") [&]() { return C->getLedgerManager().getLastClosedLedgerNum() == 2; }, 4 * simulation->getExpectedLedgerCloseTime(), false); + UNSCOPED_INFO("C reached ledger 2 after " + << std::chrono::duration_cast( + A->getClock().now() - tFeedC) + .count() + << " ms"); + // Get messages from C HerderImpl& herderC = dynamic_cast(C->getHerder()); std::vector CEnvs = herderC.getSCP().getLatestMessagesSend(2); // Pass C's messages to A and B + INFO("C sent " << CEnvs.size() << " message(s) for ledger 2; A at LCL " + << A->getLedgerManager().getLastClosedLedgerNum() + << ", B at LCL " + << B->getLedgerManager().getLastClosedLedgerNum() + << " when they are delivered"); for (auto const& env : CEnvs) { - A->getHerder().recvSCPEnvelope(env); - B->getHerder().recvSCPEnvelope(env); + INFO("C message type " + << static_cast(env.statement.pledges.type())); + auto const statusA = A->getHerder().recvSCPEnvelope(env); + auto const statusB = B->getHerder().recvSCPEnvelope(env); + UNSCOPED_INFO("delivery status A " << static_cast(statusA) + << ", B " + << static_cast(statusB)); } // Crank A and B until they're on at least ledger 3 @@ -4882,1088 +4707,6 @@ TEST_CASE("SCP message capture from previous ledger", "[herder]") REQUIRE(checkSCPHistoryEntries(C, 2, expectedTypes)); } -using Topology = std::pair, std::vector>; - -// Generate a Topology with a single org containing 3 validators of HIGH quality -static Topology -simpleThreeNode() -{ - // Generate validators - std::vector sks; - std::vector validators; - int constexpr numValidators = 3; - for (int i = 0; i < numValidators; ++i) - { - SecretKey const& key = - sks.emplace_back(SecretKey::pseudoRandomForTesting()); - ValidatorEntry& entry = validators.emplace_back(); - entry.mName = fmt::format("validator-{}", i); - entry.mHomeDomain = "A"; - entry.mQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - } - return {sks, validators}; -} - -// Generate a topology with 3 orgs of HIGH quality. Two orgs have 3 validators -// and one org has 5 validators. -static Topology -unbalancedOrgs() -{ - // Generate validators - std::vector sks; - std::vector validators; - int constexpr numValidators = 11; - for (int i = 0; i < numValidators; ++i) - { - // Orgs A and B have 3 validators each. Org C has 5 validators. - std::string org = "C"; - if (i < 3) - { - org = "A"; - } - else if (i < 6) - { - org = "B"; - } - - SecretKey const& key = - sks.emplace_back(SecretKey::pseudoRandomForTesting()); - ValidatorEntry& entry = validators.emplace_back(); - entry.mName = fmt::format("validator-{}", i); - entry.mHomeDomain = org; - entry.mQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - } - return {sks, validators}; -} - -// Generate a tier 1-like topology. This topology has 7 HIGH quality orgs, each -// with 3 validators. -static Topology -tier1Like() -{ - std::vector sks; - std::vector validators; - int constexpr numOrgs = 7; - int constexpr validatorsPerOrg = 3; - - for (int i = 0; i < numOrgs; ++i) - { - std::string const org = fmt::format("org-{}", i); - for (int j = 0; j < validatorsPerOrg; ++j) - { - SecretKey const& key = - sks.emplace_back(SecretKey::pseudoRandomForTesting()); - ValidatorEntry& entry = validators.emplace_back(); - entry.mName = fmt::format("validator-{}-{}", i, j); - entry.mHomeDomain = org; - entry.mQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - } - } - - return {sks, validators}; -} - -// Generate a slightly unbalanced topology. This topology has 7 HIGH quality -// orgs, 6 of which have 3 validators and 1 has 5 validators. -static Topology -slightlyUnbalancedOrgs() -{ - std::vector sks; - std::vector validators; - int constexpr numOrgs = 7; - - for (int i = 0; i < numOrgs; ++i) - { - std::string const org = fmt::format("org-{}", i); - int const numValidators = i == 0 ? 5 : 3; - for (int j = 0; j < numValidators; ++j) - { - SecretKey const& key = - sks.emplace_back(SecretKey::pseudoRandomForTesting()); - ValidatorEntry& entry = validators.emplace_back(); - entry.mName = fmt::format("validator-{}-{}", i, j); - entry.mHomeDomain = org; - entry.mQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - } - } - - return {sks, validators}; -} - -// Returns a random quality up to `maxQuality` -static ValidatorQuality -randomQuality(ValidatorQuality maxQuality) -{ - return static_cast(rand_uniform( - static_cast(ValidatorQuality::VALIDATOR_LOW_QUALITY), - static_cast(maxQuality))); -} - -// Returns the minimum size an org of quality `q` can have -static int constexpr minOrgSize(ValidatorQuality q) -{ - switch (q) - { - case ValidatorQuality::VALIDATOR_LOW_QUALITY: - case ValidatorQuality::VALIDATOR_MED_QUALITY: - return 1; - case ValidatorQuality::VALIDATOR_HIGH_QUALITY: - case ValidatorQuality::VALIDATOR_CRITICAL_QUALITY: - return 3; - } -} - -// Generate a random topology with up to `maxValidators` validators. Ensures at -// least one org is HIGH quality. -static Topology -randomTopology(int maxValidators) -{ - int const numValidators = rand_uniform(3, maxValidators); - int constexpr minCritOrgSize = - minOrgSize(ValidatorQuality::VALIDATOR_CRITICAL_QUALITY); - - // Generate validators - int curOrg = 0; - int curOrgSize = 0; - ValidatorQuality curQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - std::vector sks(numValidators); - std::vector validators(numValidators); - for (int i = 0; i < numValidators; ++i) - { - if (curOrgSize >= minOrgSize(curQuality) && rand_flip()) - { - // Start new org - ++curOrg; - curOrgSize = 0; - curQuality = - randomQuality(numValidators - i >= minCritOrgSize - ? ValidatorQuality::VALIDATOR_CRITICAL_QUALITY - : ValidatorQuality::VALIDATOR_MED_QUALITY); - } - - std::string const org = fmt::format("org-{}", curOrg); - SecretKey const& key = sks.at(i) = SecretKey::pseudoRandomForTesting(); - - ValidatorEntry& entry = validators.at(i); - entry.mName = fmt::format("validator-{}", i); - entry.mHomeDomain = org; - entry.mQuality = curQuality; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - - ++curOrgSize; - } - - return {sks, validators}; -} - -// Expected weight of an org with quality `orgQuality` in a topology with a max -// quality of `maxQuality` and or quality counts of `orgQualityCounts`. This -// function normalizes the weight so that the highest quality has a weight of -// `1`. -static double -expectedOrgNormalizedWeight( - std::unordered_map const& orgQualityCounts, - ValidatorQuality maxQuality, ValidatorQuality orgQuality) -{ - if (orgQuality == ValidatorQuality::VALIDATOR_LOW_QUALITY) - { - return 0.0; - } - - double normalizedWeight = 1.0; - - // For each quality level higher than `orgQuality`, divide the weight by 10 - // times the number of orgs at that quality level - for (int q = static_cast(maxQuality); q > static_cast(orgQuality); - --q) - { - normalizedWeight /= - 10 * orgQualityCounts.at(static_cast(q)); - } - return normalizedWeight; -} - -// Expected weight of a validator in an org of size `orgSize` with quality -// `orgQuality`. `maxQuality` is the maximum quality present in the -// configuration. This function normalizes the weight so that the highest -// organization-level quality has a weight of `1`. -static double -expectedNormalizedWeight( - std::unordered_map const& orgQualityCounts, - ValidatorQuality maxQuality, ValidatorQuality orgQuality, int orgSize) -{ - return expectedOrgNormalizedWeight(orgQualityCounts, maxQuality, - orgQuality) / - orgSize; -} - -// Collect information about the qualities and sizes of organizations in -// `validators` and store them in `maxQuality`, `orgQualities`, `orgSizes`, and -// `orgQualityCounts`. -static void -collectOrgInfo(ValidatorQuality& maxQuality, - std::unordered_map& orgQualities, - std::unordered_map& orgSizes, - std::unordered_map& orgQualityCounts, - std::vector const& validators) -{ - maxQuality = ValidatorQuality::VALIDATOR_LOW_QUALITY; - ValidatorQuality minQuality = ValidatorQuality::VALIDATOR_CRITICAL_QUALITY; - std::unordered_map> - orgsByQuality; - for (ValidatorEntry const& validator : validators) - { - maxQuality = std::max(maxQuality, validator.mQuality); - minQuality = std::min(minQuality, validator.mQuality); - orgQualities[validator.mHomeDomain] = validator.mQuality; - ++orgSizes[validator.mHomeDomain]; - orgsByQuality[validator.mQuality].insert(validator.mHomeDomain); - } - - // Count orgs at each quality level - for (int q = static_cast(minQuality); - q <= static_cast(maxQuality); ++q) - { - orgQualityCounts[static_cast(q)] = - orgsByQuality[static_cast(q)].size(); - if (q != static_cast(minQuality)) - { - // Add virtual org covering next lower quality level - ++orgQualityCounts[static_cast(q)]; - } - } -} - -// Given a list of validators, test that the weights of the validators herder -// reports are correct -static void -testWeights(std::vector const& validators) -{ - Config cfg = getTestConfig(0); - - cfg.generateQuorumSetForTesting(validators); - - VirtualClock clock; - Application::pointer app = createTestApplication(clock, cfg); - - // Collect info about orgs - ValidatorQuality maxQuality; - std::unordered_map orgQualities; - std::unordered_map orgSizes; - std::unordered_map orgQualityCounts; - collectOrgInfo(maxQuality, orgQualities, orgSizes, orgQualityCounts, - validators); - - // Check per-validator weights - HerderImpl& herder = dynamic_cast(app->getHerder()); - std::unordered_map normalizedOrgWeights; - for (ValidatorEntry const& validator : validators) - { - uint64_t weight = herder.getHerderSCPDriver().getNodeWeight( - validator.mKey, cfg.QUORUM_SET, false); - double normalizedWeight = - static_cast(weight) / static_cast(UINT64_MAX); - normalizedOrgWeights[validator.mHomeDomain] += normalizedWeight; - - std::string const& org = validator.mHomeDomain; - REQUIRE_THAT(normalizedWeight, - Catch::Matchers::WithinAbs( - expectedNormalizedWeight(orgQualityCounts, maxQuality, - orgQualities.at(org), - orgSizes.at(org)), - 0.0001)); - } - - // Check per-org weights - for (auto const& [org, weight] : normalizedOrgWeights) - { - REQUIRE_THAT( - weight, Catch::Matchers::WithinAbs( - expectedOrgNormalizedWeight( - orgQualityCounts, maxQuality, orgQualities.at(org)), - 0.0001)); - } -} - -// Test that HerderSCPDriver::getNodeWeight produces weights that result in a -// fair distribution of nomination wins. -TEST_CASE("getNodeWeight", "[herder]") -{ - SECTION("3 tier 1 validators, 1 org") - { - testWeights(simpleThreeNode().second); - } - - SECTION("11 tier 1 validators, 3 unbalanced orgs") - { - testWeights(unbalancedOrgs().second); - } - - SECTION("Tier1-like topology") - { - testWeights(tier1Like().second); - } - - SECTION("Tier1-like topology with a single unbalanced org") - { - testWeights(slightlyUnbalancedOrgs().second); - } - - SECTION("Random topology") - { - // Test weights for 1000 random topologies of up to 200 validators - for (int i = 0; i < 1000; ++i) - { - testWeights(randomTopology(200).second); - } - } -} - -static Value -getRandomValue() -{ - auto h = sha256(fmt::format("value {}", getGlobalRandomEngine()())); - return xdr::xdr_to_opaque(h); -} - -// A test version of NominationProtocol that exposes `updateRoundLeaders` -class TestNominationProtocol : public NominationProtocol -{ - public: - TestNominationProtocol(Slot& slot) : NominationProtocol(slot) - { - } - - std::set const& - updateRoundLeadersForTesting( - std::optional const& previousValue = std::nullopt) - { - mPreviousValue = previousValue.value_or(getRandomValue()); - updateRoundLeaders(); - return getLeaders(); - } - - // Detect fast timeouts by examining the final round number - bool - fastTimedOut() const - { - return mRoundNumber > 0; - } -}; - -// Test nomination over `numLedgers` slots. After running, check that the win -// percentages of each node and org are within 5% of the expected win -// percentages. -static void -testWinProbabilities(std::vector const& sks, - std::vector const& validators, - int const numLedgers) -{ - REQUIRE(sks.size() == validators.size()); - - // Collect info about orgs - ValidatorQuality maxQuality; - std::unordered_map orgQualities; - std::unordered_map orgSizes; - std::unordered_map orgQualityCounts; - collectOrgInfo(maxQuality, orgQualities, orgSizes, orgQualityCounts, - validators); - - // Generate a config - Config cfg = getTestConfig(); - cfg.ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING = true; - cfg.generateQuorumSetForTesting(validators); - cfg.NODE_SEED = sks.front(); - - // Create an application - VirtualClock clock; - Application::pointer app = createTestApplication(clock, cfg); - - // Run for `numLedgers` slots, recording the number of times each - // node wins nomination - UnorderedMap publishCounts; - HerderImpl& herder = dynamic_cast(app->getHerder()); - SCP& scp = herder.getSCP(); - int fastTimeouts = 0; - for (int i = 0; i < numLedgers; ++i) - { - auto s = std::make_shared(i, scp); - TestNominationProtocol np(*s); - - std::set const& leaders = np.updateRoundLeadersForTesting(); - REQUIRE(leaders.size() == 1); - for (NodeID const& leader : leaders) - { - ++publishCounts[leader]; - } - - if (np.fastTimedOut()) - { - ++fastTimeouts; - } - } - - CLOG_INFO(Herder, "Fast Timeouts: {} ({}%)", fastTimeouts, - fastTimeouts * 100.0 / numLedgers); - - // Compute total expected normalized weight across all nodes - double totalNormalizedWeight = 0.0; - for (ValidatorEntry const& validator : validators) - { - totalNormalizedWeight += - expectedNormalizedWeight(orgQualityCounts, maxQuality, - orgQualities.at(validator.mHomeDomain), - orgSizes.at(validator.mHomeDomain)); - } - - // Check validator win rates - std::map orgPublishCounts; - for (ValidatorEntry const& validator : validators) - { - NodeID const& nodeID = validator.mKey; - int publishCount = publishCounts[nodeID]; - - // Compute and report node's win rate - double winRate = static_cast(publishCount) / numLedgers; - CLOG_INFO(Herder, "Node {} win rate: {} (published {} ledgers)", - cfg.toShortString(nodeID), winRate, publishCount); - - // Expected win rate is `weight / total weight` - double expectedWinRate = - expectedNormalizedWeight(orgQualityCounts, maxQuality, - orgQualities.at(validator.mHomeDomain), - orgSizes.at(validator.mHomeDomain)) / - totalNormalizedWeight; - - // Check that actual win rate is within .05 of expected win - // rate. - REQUIRE_THAT(winRate, - Catch::Matchers::WithinAbs(expectedWinRate, 0.05)); - - // Record org publish counts for the next set of checks - orgPublishCounts[validator.mHomeDomain] += publishCount; - } - - // Check org win rates - for (auto const& [org, count] : orgPublishCounts) - { - // Compute and report org's win rate - double winRate = static_cast(count) / numLedgers; - CLOG_INFO(Herder, "Org {} win rate: {} (published {} ledgers)", org, - winRate, count); - - // Expected win rate is `weight / total weight` - double expectedWinRate = - expectedOrgNormalizedWeight(orgQualityCounts, maxQuality, - orgQualities.at(org)) / - totalNormalizedWeight; - - // Check that actual win rate is within .05 of expected win - // rate. - REQUIRE_THAT(winRate, - Catch::Matchers::WithinAbs(expectedWinRate, 0.05)); - } -} - -// Test that the nomination algorithm produces a fair distribution of ledger -// publishers. -TEST_CASE("Fair nomination win rates", "[herder]") -{ - SECTION("3 tier 1 validators, 1 org") - { - auto [sks, validators] = simpleThreeNode(); - testWinProbabilities(sks, validators, 10000); - } - - SECTION("11 tier 1 validators, 3 unbalanced orgs") - { - auto [sks, validators] = unbalancedOrgs(); - testWinProbabilities(sks, validators, 10000); - } - - SECTION("Tier 1-like topology") - { - auto [sks, validators] = tier1Like(); - testWinProbabilities(sks, validators, 10000); - } - - SECTION("Tier 1-like topology with a single unbalanced org") - { - auto [sks, validators] = slightlyUnbalancedOrgs(); - testWinProbabilities(sks, validators, 10000); - } - - SECTION("Random topology") - { - for (int i = 0; i < 10; ++i) - { - auto [sks, validators] = randomTopology(50); - testWinProbabilities(sks, validators, 10000); - } - } -} - -namespace -{ -// Returns a new `Topology` with the last org in `t` replaced with a new org -// with 3 validators. Requires that the last org in `t` have 3 validators and be -// contiguous at the back of the validators vecto. -Topology -replaceOneOrg(Topology const& t) -{ - Topology t2(t); // Copy the topology - auto& [sks, validators] = t2; - REQUIRE(sks.size() == validators.size()); - - // Give the org a unique name - std::string const orgName = "org-replaced"; - - // Double check that the new org name is unique - for (ValidatorEntry const& v : validators) - { - REQUIRE(v.mHomeDomain != orgName); - } - - // Remove the last org - constexpr int validatorsPerOrg = 3; - sks.resize(sks.size() - validatorsPerOrg); - validators.resize(validators.size() - validatorsPerOrg); - - // Add new org with 3 validators - int constexpr numValidators = 3; - for (int j = 0; j < numValidators; ++j) - { - SecretKey const& key = - sks.emplace_back(SecretKey::pseudoRandomForTesting()); - ValidatorEntry& entry = validators.emplace_back(); - entry.mName = fmt::format("validator-replaced-{}", j); - entry.mHomeDomain = orgName; - entry.mQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - } - - return {sks, validators}; -} - -// Add `orgsToAdd` new orgs to the topology `t`. Each org will have 3 -// validators. -Topology -addOrgs(int orgsToAdd, Topology const& t) -{ - Topology t2(t); // Copy the topology - auto& [sks, validators] = t2; - REQUIRE(sks.size() == validators.size()); - - // Generate new orgs - for (int i = 0; i < orgsToAdd; ++i) - { - std::string const org = fmt::format("new-org-{}", i); - int constexpr numValidators = 3; - for (int j = 0; j < numValidators; ++j) - { - SecretKey const& key = - sks.emplace_back(SecretKey::pseudoRandomForTesting()); - ValidatorEntry& entry = validators.emplace_back(); - entry.mName = fmt::format("new-validator-{}-{}", i, j); - entry.mHomeDomain = org; - entry.mQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - } - } - return t2; -} - -// Returns `true` if the set intersection of `leaders1` and `leaders2` is not -// empty. -bool -leadersIntersect(std::set const& leaders1, - std::set const& leaders2) -{ - std::vector intersection; - std::set_intersection(leaders1.begin(), leaders1.end(), leaders2.begin(), - leaders2.end(), std::back_inserter(intersection)); - return !intersection.empty(); -} - -// Given two quorum sets consisting of validators in `validators1` and -// `validators2`, this function returns the probability that the two quorum sets -// will agree on a leader in the first round of nomination. -double -computeExpectedFirstRoundAgreementProbability( - std::vector const& validators1, - std::vector const& validators2) -{ - // Gather orgs - std::set orgs1; - std::transform(validators1.begin(), validators1.end(), - std::inserter(orgs1, orgs1.end()), - [](ValidatorEntry const& v) { return v.mHomeDomain; }); - std::set orgs2; - std::transform(validators2.begin(), validators2.end(), - std::inserter(orgs2, orgs2.end()), - [](ValidatorEntry const& v) { return v.mHomeDomain; }); - - // Compute overlap - std::vector sharedOrgs; - std::set_intersection(orgs1.begin(), orgs1.end(), orgs2.begin(), - orgs2.end(), std::back_inserter(sharedOrgs)); - - // Probability of agreement in first round is (orgs overlapping / orgs1) * - // (orgs overlapping / orgs2). That's the probability that the two sides - // will pick any overlapping org. The algorithm guarantees that if they pick - // overlapping validator, they'll pick the same validator. - double overlap = static_cast(sharedOrgs.size()); - return overlap / orgs1.size() * overlap / orgs2.size(); -} - -// Test that the nomination algorithm behaves as expected when the two quorum -// sets `qs1` and `qs2` are not equivalent. This function requires that both -// quorum sets overlap, and contain only a single quality level of validators. -// Runs simulation for `numLedgers` slots. -// NOTE: This test counts any failure to agree on a leader as a timeout. In -// practice, it's possible that one side of the split is large enough to proceed -// without the other side. In this case, the larger side might not experience a -// timeout and "drag" the other side through consensus with it. However, this -// test aims to analyze the worst case scenario where the two sides are fairly -// balanced and real-world networking conditions are in place (some nodes -// lagging, etc), such that disagreement always results in a timeout. -void -testAsymmetricTimeouts(Topology const& qs1, Topology const& qs2, - int const numLedgers) -{ - auto const& [sks1, validators1] = qs1; - auto const& [sks2, validators2] = qs2; - - REQUIRE(sks1.size() == validators1.size()); - REQUIRE(sks2.size() == validators2.size()); - - // Generate configs and nodes representing one validator with each quorum - // set - std::vector clocks(2); - std::vector apps; - for (int i = 0; i < 2; ++i) - { - Config cfg = getTestConfig(i); - cfg.ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING = true; - cfg.generateQuorumSetForTesting(i == 0 ? validators1 : validators2); - cfg.NODE_SEED = i == 0 ? sks1.back() : sks2.back(); - - auto app = apps.emplace_back(createTestApplication(clocks.at(i), cfg)); - } - - // Run the nomination algorithm for `numLedgers` slots. Simulate timeouts by - // re-running slots that don't agree on a leader until their leader - // elections overlap. Record the number of timeouts it takes for the two - // quorum sets to agree on a leader in `timeouts`, which is effectively a - // mapping from number of timeouts to the number of ledgers that experienced - // that many timeouts. - std::vector timeouts(std::max(validators1.size(), validators2.size())); - for (int i = 0; i < numLedgers; ++i) - { - Value const v = getRandomValue(); - SCP& scp1 = dynamic_cast(apps.at(0)->getHerder()).getSCP(); - SCP& scp2 = dynamic_cast(apps.at(1)->getHerder()).getSCP(); - auto s1 = std::make_shared(i, scp1); - auto s2 = std::make_shared(i, scp2); - - TestNominationProtocol np1(*s1); - TestNominationProtocol np2(*s2); - - for (int j = 0; j < timeouts.size(); ++j) - { - std::set const& leaders1 = - np1.updateRoundLeadersForTesting(v); - std::set const& leaders2 = - np2.updateRoundLeadersForTesting(v); - REQUIRE(leaders1.size() == j + 1); - REQUIRE(leaders2.size() == j + 1); - - if (leadersIntersect(leaders1, leaders2)) - { - // Agreed on a leader! Record the number of timeouts resulted. - ++timeouts.at(j); - break; - } - } - - // If leaders don't intersect after running through the loop then the - // two quorum sets have no overlap and the test is broken. - REQUIRE(leadersIntersect(np1.getLeaders(), np2.getLeaders())); - } - - // For the first round, we can easily compute the expected agreement - // probability. For subsequent rounds, we check only that the success rate - // increases over time (modulo some small epsilon). - double expectedSuccessRate = - computeExpectedFirstRoundAgreementProbability(validators1, validators2); - - // Allow for some small decrease in success rate from the theoretical value. - // We're working with probabilistic simulation here so we can't be too - // strict or the test will be flaky. - double constexpr epsilon = 0.1; - - // There's not enough data in the tail of the distribution to allow us to - // assert that the success rate is what's expected. To avoid sporadic test - // failures, we cut off `tailCutoffPoint` of the tail of the distribution - // for the purposes of asserting test values. However, the test will still - // log those success rates for manual examination. - double constexpr tailCutoffPoint = 0.05; - - int numLedgersRemaining = numLedgers; - for (int i = 0; i < timeouts.size(); ++i) - { - int const numTimeouts = timeouts.at(i); - if (numTimeouts == 0) - { - // Avoid cluttering output - continue; - } - - CLOG_INFO(Herder, "Ledgers with {} timeouts: {} ({}%)", i, numTimeouts, - static_cast(numTimeouts) * 100 / numLedgers); - - if (numLedgersRemaining > numLedgers * tailCutoffPoint) - { - // Check that success rate increases over time. Allow some epsilon - // decrease because this is a probabilistic simulation. Also stop - // checking when we're at the last `tailCutoffPoint` timeouts as the - // data is too sparse to be useful. - double successRate = - static_cast(timeouts.at(i)) / numLedgersRemaining; - REQUIRE(successRate > expectedSuccessRate - epsilon); - - // Take max of success rate and previous success rate to avoid - // accidentally accepting a declining success rate due to episilon. - expectedSuccessRate = std::max(successRate, expectedSuccessRate); - numLedgersRemaining -= numTimeouts; - } - } -} -} // namespace - -// Test timeouts with asymmetric quorums. This test serves two purposes: -// 1. It contains assertions checking for moderate (10%) deviations from the -// expected behavior of the nomination algorithm. These should detect any -// major issues/regressions with the algorithm. -// 2. It logs the distributions of timeouts for manual inspection. This is -// useful for understanding the behavior of the algorithm and for testing -// specific scenarios one might be interested in (e.g., if tier 1 disagrees -// on one org's presence in tier 1, what is the impact on nomination -// timeouts?). -// NOTE: This provides a worst-case analysis of timeouts. See the NOTE on -// `testAsymmetricTimeouts` for more details. -TEST_CASE("Asymmetric quorum timeouts", "[herder]") -{ - // Number of slots to run for - int constexpr numLedgers = 20000; - - SECTION("Tier 1-like topology with replaced org") - { - auto t = tier1Like(); - testAsymmetricTimeouts(t, replaceOneOrg(t), numLedgers); - } - - SECTION("Tier 1-like topology with 1 added org") - { - auto t = tier1Like(); - testAsymmetricTimeouts(t, addOrgs(1, t), numLedgers); - } - - SECTION("Tier 1-like topology with 3 added orgs") - { - auto t = tier1Like(); - testAsymmetricTimeouts(t, addOrgs(3, t), numLedgers); - } -} - -// Test that the nomination algorithm behaves as expected when a random -// `numUnresponsive` set of nodes in `qs` are unresponsive. Runs simulation for -// `numLedgers` slots. -static void -testUnresponsiveTimeouts(Topology const& qs, int numUnresponsive, - int const numLedgers) -{ - auto const& [sks, validators] = qs; - REQUIRE(sks.size() == validators.size()); - REQUIRE(numUnresponsive < validators.size()); - - // extract and shuffle node ids. Choose `numUnresponsive` nodes to be the - // unresponsive nodes. - std::vector nodeIDs; - std::transform(validators.begin(), validators.end(), - std::back_inserter(nodeIDs), - [](ValidatorEntry const& v) { return v.mKey; }); - stellar::shuffle(nodeIDs.begin(), nodeIDs.end(), getGlobalRandomEngine()); - std::set unresponsive(nodeIDs.begin(), - nodeIDs.begin() + numUnresponsive); - - // Collect info about orgs - ValidatorQuality maxQuality; - std::unordered_map orgQualities; - std::unordered_map orgSizes; - std::unordered_map orgQualityCounts; - collectOrgInfo(maxQuality, orgQualities, orgSizes, orgQualityCounts, - validators); - - // Compute total weight of all validators, as well as the total weight of - // unresponsive validators - double totalWeight = 0.0; - double unresponsiveWeight = 0.0; - for (ValidatorEntry const& validator : validators) - { - double normalizedWeight = - expectedNormalizedWeight(orgQualityCounts, maxQuality, - orgQualities.at(validator.mHomeDomain), - orgSizes.at(validator.mHomeDomain)); - totalWeight += normalizedWeight; - if (unresponsive.count(validator.mKey)) - { - unresponsiveWeight += normalizedWeight; - } - } - - // Compute the average weight of an unresponsive node - double avgUnresponsiveWeight = unresponsiveWeight / numUnresponsive; - - // Compute expected number of ledgers experiencing `n` timeouts where `n` is - // the index of the `timeouts` vector. This vector is a mapping from number - // of timeouts to expected number of ledgers experiencing that number of - // timeouts. - std::vector expectedTimeouts(numUnresponsive + 1); - double remainingWeight = totalWeight; - int remainingUnresponsive = numUnresponsive; - int remainingLedgers = numLedgers; - for (int i = 0; i < expectedTimeouts.size(); ++i) - { - double timeoutProb = - (avgUnresponsiveWeight * remainingUnresponsive) / remainingWeight; - // To get expected number of ledgers experiencing `i` timeouts, we take - // the probability a timeout does not occur and multiply it by the - // number of remaining ledgers. - int expectedLedgers = (1 - timeoutProb) * remainingLedgers; - expectedTimeouts.at(i) = expectedLedgers; - - // Remaining ledgers decreases by expected number of ledgers - // experiencing `i` timeouts - remainingLedgers -= expectedLedgers; - - // For `i+1` timeouts to occur, an unresponsive node must be chosen. - // Therefore, deduct the average weight of an unresponsive node from the - // total weight left in the network. - remainingWeight -= avgUnresponsiveWeight; - --remainingUnresponsive; - } - - // Generate a config - Config cfg = getTestConfig(); - cfg.ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING = true; - cfg.generateQuorumSetForTesting(validators); - cfg.NODE_SEED = sks.front(); - - // Create an application - VirtualClock clock; - Application::pointer app = createTestApplication(clock, cfg); - - // Run for `numLedgers` slots, recording the number of times each slot timed - // out due to unresponsive nodes before successfully electing a responsive - // leader. - SCP& scp = dynamic_cast(app->getHerder()).getSCP(); - std::vector timeouts(numUnresponsive + 1); - for (int i = 0; i < numLedgers; ++i) - { - Value const v = getRandomValue(); - auto s = std::make_shared(i, scp); - - TestNominationProtocol np(*s); - for (int i = 0; i < timeouts.size(); ++i) - { - std::set const& leaders = - np.updateRoundLeadersForTesting(v); - // If leaders is a subset of unresponsive, then a timeout occurs. - if (!std::includes(unresponsive.begin(), unresponsive.end(), - leaders.begin(), leaders.end())) - { - ++timeouts.at(i); - break; - } - } - } - - // Allow for some small multiplicative increase in timeouts from the - // theoretical value. We're working with probabilistic simulation here so - // we can't be too strict or the test will be flaky. - double constexpr epsilon = 1.1; - - // There's not enough data in the tail of the distribution to allow us to - // assert that the timeout values are what's expected. To avoid sporadic - // test failures, we cut off `tailCutoffPoint` of the tail of the - // distribution for the purposes of asserting test values. However, the test - // will still log those values for manual examination. - double constexpr tailCutoffPoint = 0.05; - - // Analyze timeouts - int numLedgersRemaining = numLedgers; - for (int i = 0; i < timeouts.size(); ++i) - { - int const numTimeouts = timeouts.at(i); - int const expectedNumTimeouts = expectedTimeouts.at(i); - - if (numLedgersRemaining > numLedgers * tailCutoffPoint) - { - // Check that timeouts are less than epsilon times the expected - // value. Also stop checking when we're at the last - // `tailCutoffPoint` timeouts as the data is too sparse to be - // useful. - REQUIRE(numTimeouts < expectedNumTimeouts * epsilon); - } - CLOG_INFO(Herder, "Ledgers with {} timeouts: {} ({}%)", i, numTimeouts, - numTimeouts * 100.0 / numLedgers); - numLedgersRemaining -= numTimeouts; - } -} - -// Test timeouts for a tier 1-like topology with 1-5 unresponsive nodes. This -// test serves two purposes: -// 1. It contains assertions checking for moderate (10%) deviations from the -// expected behavior of the nomination algorithm. These should detect any -// major issues/regressions with the algorithm. -// 2. It logs the distributions of timeouts for manual inspection. This is -// useful for understanding the behavior of the algorithm and for testing -// specific scenarios one might be interested in (e.g., if 3 tier 1 nodes -// are heavily lagging, what is the impact on nomination timeouts?). -TEST_CASE("Unresponsive quorum timeouts", "[herder]") -{ - // Number of slots to run for - int constexpr numLedgers = 20000; - - auto t = tier1Like(); - for (int i = 1; i <= 5; ++i) - { - CLOG_INFO(Herder, "Simulating nomination with {} unresponsive nodes", - i); - testUnresponsiveTimeouts(t, i, numLedgers); - } -} - -TEST_CASE("nomination timeouts with partial upgrade arming", - "[herder][acceptance]") -{ - // Configure simulation to use automatic quorum set configuration so that it - // runs with the application-specific leader election algorithm, which does - // not introduce its own timeouts. - Hash networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); - Simulation::pointer simulation = - Topologies::separateAllHighQuality(16, networkID, [&](int i) { - return getTestConfig(i, Config::TESTDB_DEFAULT); - }); - simulation->fullyConnectAllPending(); - simulation->startAllNodes(); - auto nodes = simulation->getNodes(); - REQUIRE(nodes.size() == 16); - - // Let the network run for a few ledgers normally first - auto const expectedLedgerCloseTime = - simulation->getExpectedLedgerCloseTime(); - simulation->crankUntil( - [&]() { return simulation->haveAllExternalized(5, 1); }, - 10 * expectedLedgerCloseTime, false); - - // Create an upgrade to arm on a subset of nodes. - Upgrades::UpgradeParameters scheduledUpgrades; - auto lclCloseTime = - VirtualClock::from_time_t(nodes[0] - ->getLedgerManager() - .getLastClosedLedgerHeader() - .header.scpValue.closeTime); - - // Set upgrade time to now so it's active immediately - scheduledUpgrades.mUpgradeTime = lclCloseTime; - - // Upgrade the base fee by a small amount - auto const currentFee = nodes[0]->getLedgerManager().getLastTxFee(); - scheduledUpgrades.mBaseFee = currentFee + 100; - - // Limit max timeouts per slot to 1 - constexpr uint32_t maxTimeouts = 1; - scheduledUpgrades.mNominationTimeoutLimit = maxTimeouts; - - // Reduce upgrade window to 4 minutes - constexpr std::chrono::minutes upgradeWindow(4); - scheduledUpgrades.mExpirationMinutes = upgradeWindow; - - // Number of ledgers to check timeouts during - constexpr int ledgersToRun = 20; - - // Maximum total timeout duration for the test. Worst case is that each slot - // experiences 1 timeout, which adds 1 second each. - constexpr auto maxTotalTimeoutDuration = std::chrono::seconds(ledgersToRun); - - // Ensure upgrade window is set properly so that the upgrade doesn't expire - // during the `ledgersToRun` time period - REQUIRE(upgradeWindow > - expectedLedgerCloseTime * ledgersToRun + maxTotalTimeoutDuration); - - // Arm upgrades on 10 nodes (just 1 shy of a quorum) - for (size_t i = 0; i < 10; ++i) - { - nodes[i]->getHerder().setUpgrades(scheduledUpgrades); - } - - // Track initial ledger number - auto const startLedger = - nodes[0]->getLedgerManager().getLastClosedLedgerNum(); - - // Run for `ledgersToRun` more ledgers with mixed upgrade state - auto& herder = dynamic_cast(nodes[0]->getHerder()); - HerderSCPDriver const& driver = herder.getHerderSCPDriver(); - for (int i = 1; i <= ledgersToRun; ++i) - { - uint32_t const ledger = startLedger + i; - simulation->crankUntil( - [&]() { return simulation->haveAllExternalized(ledger, 1); }, - expectedLedgerCloseTime * 2, false); - - // Should see at most `maxTimeouts` per slot, depending on the round - // leaders. - std::optional timeouts = driver.getNominationTimeouts(ledger); - REQUIRE(timeouts.has_value()); - REQUIRE(timeouts.value() <= maxTimeouts); - } - - // Helper to check whether upgrade is still active by comparing with - // `scheduledUpgrades` - std::string const upgradeJson = scheduledUpgrades.toJson(); - auto const upgradeIsActive = [&]() { - return herder.getUpgrades().getParameters().toJson() == upgradeJson; - }; - - // Verify upgrade is still active - REQUIRE(upgradeIsActive()); - - // Verify that upgrade expires properly after the window - simulation->crankUntil(std::not_fn(upgradeIsActive), upgradeWindow, false); - - // Ensure the changed fields are all reset - auto const& upgradeParams = herder.getUpgrades().getParameters(); - REQUIRE(!upgradeParams.mBaseFee.has_value()); - REQUIRE(!upgradeParams.mNominationTimeoutLimit.has_value()); - REQUIRE(!upgradeParams.mExpirationMinutes.has_value()); - - // Verify the upgrade did not go through - REQUIRE(nodes[0]->getLedgerManager().getLastTxFee() == currentFee); -} - TEST_CASE("late joining node reaches consensus", "[herder]") { auto networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); @@ -6100,7 +4843,7 @@ TEST_CASE_VERSIONS("Herder properly validates when tx set is missing", }); } -// One of four validators nominates a non-canonical empty tx set (a tx set +// One of four validators proposes a non-canonical empty tx set (a tx set // without transactions that isn't encoded as `TxSetXDRFrame::makeEmpty`, // which history can't replay) on every ledger. Before the protocol gate the // network externalizes it, after the gate it's rejected as invalid. @@ -6132,8 +4875,7 @@ TEST_CASE("network does not externalize non-canonical empty tx set", auto simulation = Topologies::core( 4, 0.75, sha256(getTestConfig().NETWORK_PASSPHRASE), makeConfig); auto nodes = simulation->getNodes(); - static_cast(nodes[0]->getHerder()) - .mNominatedTxSetOverride = + static_cast(nodes[0]->getHerder()).mProposedTxSetOverride = [&](LedgerHeaderHistoryEntry const& lcl) { return makeNonCanonicalEmptyTxSet(lcl.hash); }; @@ -6169,8 +4911,11 @@ TEST_CASE("network does not externalize non-canonical empty tx set", } else { + // Honest nodes do not vote for the bad proposal. There is no + // nomination to carry it into their ballots (where it used to be + // replaced with an empty-tx-set value): the round times out and + // the next leader's value closes the ledger. REQUIRE(externalized == 0); - REQUIRE(replaced > 0); } }; @@ -6197,7 +4942,7 @@ TEST_CASE("network externalizes empty-tx-set on missing value", "[herder][tx]") auto cfg = getTestConfig(i, Config::TESTDB_DEFAULT); cfg.EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD = true; cfg.TX_SET_DOWNLOAD_TIMEOUT = std::chrono::milliseconds{0}; - cfg.TESTING_NOMINATE_RANDOM_VALUES = true; + cfg.TESTING_PROPOSE_RANDOM_VALUES = true; return cfg; }); @@ -6301,7 +5046,7 @@ TEST_CASE("SCP state restore with missing tx set", "[herder]") // The restored value's tx set is missing and nothing is fetching it, but // the value is still structurally valid - REQUIRE(driver.validateValue(slot, value, /*nomination*/ false) == + REQUIRE(driver.validateValue(slot, value, /*proposal*/ false) == SCPDriver::kStructurallyValidValue); // The peer's view of the slot: it timed out waiting for the missing tx @@ -6372,11 +5117,10 @@ triggerTimerProtocolSupported() } // Four top-tier validators over TCP on the real clock, with a 1s artificial -// apply delay and a configurable nomination-emit delay so we can actually see -// the impact of the two different timers. +// apply delay. static Simulation::pointer makeTriggerTimerSimulation( - bool forcePrepareStartTimer, std::chrono::milliseconds nominationEmitDelay, + bool forcePrepareStartTimer, std::chrono::milliseconds driftClockOffset = std::chrono::milliseconds::zero(), std::optional startingProtocol = std::nullopt) @@ -6391,8 +5135,6 @@ makeTriggerTimerSimulation( forcePrepareStartTimer; cfg.ARTIFICIALLY_DELAY_LEDGER_CLOSE_FOR_TESTING = std::chrono::milliseconds(1000); - cfg.ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING = - nominationEmitDelay; // Remember enough SCP slots that the tests can attribute every // externalized value in their measurement windows to its // proposer. @@ -6469,7 +5211,6 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") int64_t driftedNodeFallbacks{0}; int64_t maxOtherNodeFallbacks{0}; int64_t driftedLedSlots{0}; - bool sawNominationTimeout{false}; }; auto fallbackCount = [](Application::pointer const& app) { @@ -6487,11 +5228,10 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") auto runSimulation = [&](bool forcePrepareStartTimer, - std::chrono::milliseconds nominationEmitDelay, std::chrono::milliseconds triggerClockOffset = std::chrono::milliseconds::zero()) -> RunResult { - auto simulation = makeTriggerTimerSimulation( - forcePrepareStartTimer, nominationEmitDelay, triggerClockOffset); + auto simulation = makeTriggerTimerSimulation(forcePrepareStartTimer, + triggerClockOffset); auto nodes = simulation->getNodes(); std::vector fallbackCounts; @@ -6524,18 +5264,6 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") result.maxOtherNodeFallbacks = std::max(result.maxOtherNodeFallbacks, delta); } - - auto const& driver = - dynamic_cast(nodes[i]->getHerder()) - .getHerderSCPDriver(); - for (uint32_t ledger = startLedger + 1; ledger <= targetLedger; - ++ledger) - { - auto timeouts = driver.getNominationTimeouts(ledger); - result.sawNominationTimeout = - result.sawNominationTimeout || - (timeouts.has_value() && timeouts.value() > 0); - } } // Count the externalized values the drifted @@ -6580,13 +5308,10 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") return result; }; - // New timer is faster without drift. + // Without drift the new timer never falls back. (There is no nomination + // phase whose duration would separate the two anchors any more.) { - auto const nominationDelay = std::chrono::milliseconds(1000); - auto const oldTimer = runSimulation(true, nominationDelay); - auto const newTimer = runSimulation(false, nominationDelay); - - REQUIRE(newTimer.elapsed < oldTimer.elapsed); + auto const newTimer = runSimulation(false); REQUIRE(newTimer.totalFallbacks == 0); } @@ -6594,8 +5319,7 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") // One node drifting ahead falls back. { - auto const nodeAhead = runSimulation( - false, std::chrono::milliseconds::zero(), driftOffset); + auto const nodeAhead = runSimulation(false, driftOffset); REQUIRE(nodeAhead.driftedNodeFallbacks >= LEDGERS_TO_RUN - nodeAhead.driftedLedSlots - FALLBACK_SLACK); @@ -6607,8 +5331,7 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") // One node drifting behind falls back. { - auto const nodeBehind = runSimulation( - false, std::chrono::milliseconds::zero(), -driftOffset); + auto const nodeBehind = runSimulation(false, -driftOffset); REQUIRE(nodeBehind.driftedNodeFallbacks >= LEDGERS_TO_RUN - nodeBehind.driftedLedSlots - FALLBACK_SLACK); @@ -6617,122 +5340,11 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") REQUIRE(nodeBehind.maxOtherNodeFallbacks <= nodeBehind.driftedLedSlots + FALLBACK_SLACK); } - - // Long nomination does not cause timer fallback - { - auto const nominationDelay = std::chrono::milliseconds(5000); - auto const slowNomination = runSimulation(false, nominationDelay); - REQUIRE(slowNomination.sawNominationTimeout); - REQUIRE(slowNomination.totalFallbacks == 0); - } -} - -TEST_CASE("trigger timer switches anchor at protocol 28 upgrade", - "[herder][upgrades][!hide]") -{ - if (!triggerTimerProtocolSupported()) - { - return; - } - - auto const upgradeVersion = - static_cast(CONSENSUS_CLOSE_TIME_TRIGGER_PROTOCOL_VERSION); - - // With a delayed nomination emit, the prepare-start timer paces ledgers - // at roughly expectedClose + nominationEmitDelay (nomination happens - // before the anchor point), while the consensus-close-time timer absorbs - // the nomination delay and paces at expectedClose. This delta helps us - // measure the timer change after the upgrade. - constexpr uint32_t LEDGERS_TO_MEASURE = 8; - auto const nominationEmitDelay = std::chrono::milliseconds(1000); - - struct RunResult - { - std::chrono::milliseconds preUpgrade; - std::chrono::milliseconds postUpgrade; - }; - - auto runSimulation = [&](bool forcePrepareStartTimer) -> RunResult { - // Start the network one protocol before the trigger-timer switch. - auto simulation = makeTriggerTimerSimulation( - forcePrepareStartTimer, nominationEmitDelay, - std::chrono::milliseconds::zero(), upgradeVersion - 1); - auto nodes = simulation->getNodes(); - auto const expectedClose = simulation->getExpectedLedgerCloseTime(); - - auto lclVersion = [](Application::pointer const& node) { - return node->getLedgerManager() - .getLastClosedLedgerHeader() - .header.ledgerVersion; - }; - - // Measure the cadence on the pre-28 protocol. - simulation->crankUntil( - [&]() { return simulation->haveAllExternalized(3, 1); }, - 10 * expectedClose, false); - auto const preUpgrade = closeLedgers(simulation, LEDGERS_TO_MEASURE); - for (auto const& node : nodes) - { - REQUIRE(lclVersion(node) == upgradeVersion - 1); - } - - // Upgrade to protocol 28 - Upgrades::UpgradeParameters scheduledUpgrades; - scheduledUpgrades.mUpgradeTime = - VirtualClock::from_time_t(nodes[0] - ->getLedgerManager() - .getLastClosedLedgerHeader() - .header.scpValue.closeTime); - scheduledUpgrades.mProtocolVersion = upgradeVersion; - for (auto const& node : nodes) - { - node->getHerder().setUpgrades(scheduledUpgrades); - } - - // Crank until every node has closed the upgrade ledger, then one - // more ledger so the measured window is fully post-upgrade. - simulation->crankUntil( - [&]() { - return std::all_of(nodes.begin(), nodes.end(), - [&](Application::pointer const& node) { - return lclVersion(node) == - upgradeVersion; - }); - }, - 10 * expectedClose, false); - closeLedgers(simulation, 1); - - auto const postUpgrade = closeLedgers(simulation, LEDGERS_TO_MEASURE); - for (auto const& node : nodes) - { - REQUIRE(lclVersion(node) == upgradeVersion); - } - - return {preUpgrade, postUpgrade}; - }; - - // Expected cadence saving is nominationEmitDelay per ledger; splitting - // pass/fail at half of it tolerates scheduling noise in both directions. - auto const cadenceMargin = LEDGERS_TO_MEASURE * nominationEmitDelay / 2; - - // Crossing the boundary switches to the consensus-close-time anchor: - // post-upgrade ledgers close significantly faster. - { - auto const result = runSimulation(false); - REQUIRE(result.postUpgrade + cadenceMargin < result.preUpgrade); - } - - // The override flag keeps the prepare-start anchor after the upgrade, so - // the cadence does not improve. - { - auto const result = runSimulation(true); - REQUIRE(result.postUpgrade + cadenceMargin > result.preUpgrade); - } } namespace stellar { -class EarlyNominationTestAccess +class LeaderBuildTestAccess { public: static VirtualClock::time_point @@ -6843,13 +5455,12 @@ class EarlyNominationTestAccess SecretKey::fromSeed(sha256(fmt::format("absent-peer-{}", i))) .getPublicKey()}; scp.updateLocalQuorumSet(qset); - if (scp.predictNominationLeaders( - lcl.header.ledgerSeq + 1, - xdr::xdr_to_opaque(lcl.header.scpValue), 1) - .count(scp.getLocalNodeID())) + herder.mPendingEnvelopes.addSCPQuorumSet( + scp.getLocalNode()->getQuorumSetHash(), qset); + herder.mPendingEnvelopes.forceRebuildQuorum(); + if (herder.getHerderSCPDriver().isLocalNodeRoundLeader( + lcl.header.ledgerSeq + 1, 1)) { - herder.mPendingEnvelopes.addSCPQuorumSet( - scp.getLocalNode()->getQuorumSetHash(), qset); return; } } @@ -6858,7 +5469,7 @@ class EarlyNominationTestAccess }; } -TEST_CASE("trigger fallback credits elapsed time through nomination", +TEST_CASE("trigger fallback credits elapsed time up to the first ballot", "[herder][trigger-construction]") { using namespace std::chrono; @@ -6872,23 +5483,23 @@ TEST_CASE("trigger fallback credits elapsed time through nomination", int expectedWaitMs; int expectedFallbacks; }; - auto scenario = - GENERATE(Scenario{350, 1900, 2100, 0, true, 0, 0}, - Scenario{350, 1900, 2100, 4000, true, 1800, 1}, - Scenario{350, 1900, 2100, -4000, true, 1800, 1}, - Scenario{350, 1900, 4100, 0, true, 0, 1}, - Scenario{350, 1900, 2100, 0, false, 1800, 1}, - Scenario{350, 1500, 1700, 0, true, 300, 0}, - // A slow first round need not execute a timeout callback - // before entering ballot. Its elapsed time is still real. - Scenario{100, 2000, 2250, 0, true, 0, 0}, - // Followers need no construction, but still wait on peers. - Scenario{0, 2250, 2450, 0, true, 0, 0}, - // Delayed timer callbacks must not hide elapsed time. - Scenario{100, 6500, 6700, 0, true, 0, 0}, - // Long nomination does not excuse arbitrary clock drift. - Scenario{100, 6500, 6700, 10000, true, 1800, 1}, - Scenario{100, 6500, 6700, -10000, true, 1800, 1}); + auto scenario = GENERATE( + Scenario{350, 1900, 2100, 0, true, 0, 0}, + Scenario{350, 1900, 2100, 4000, true, 1800, 1}, + Scenario{350, 1900, 2100, -4000, true, 1800, 1}, + Scenario{350, 1900, 4100, 0, true, 0, 1}, + Scenario{350, 1900, 2100, 0, false, 1800, 1}, + Scenario{350, 1500, 1700, 0, true, 300, 0}, + // A slow first round need not execute a timeout callback + // before entering ballot. Its elapsed time is still real. + Scenario{100, 2000, 2250, 0, true, 0, 0}, + // Followers need no construction, but still wait on peers. + Scenario{0, 2250, 2450, 0, true, 0, 0}, + // Delayed timer callbacks must not hide elapsed time. + Scenario{100, 6500, 6700, 0, true, 0, 0}, + // A long wait for a proposal does not excuse arbitrary clock drift. + Scenario{100, 6500, 6700, 10000, true, 1800, 1}, + Scenario{100, 6500, 6700, -10000, true, 1800, 1}); CAPTURE(scenario.buildMs, scenario.closeMs, scenario.clockOffsetMs, scenario.recordTrigger); VirtualClock clock; @@ -6907,15 +5518,14 @@ TEST_CASE("trigger fallback credits elapsed time through nomination", herder.setTrackingSCPState(slot, value, true); if (scenario.recordTrigger) { - driver.recordNominationTrigger(slot); + driver.recordTrigger(slot); } clock.setCurrentVirtualTime(origin + milliseconds(scenario.buildMs)); - driver.recordSCPEvent(slot, true); clock.setCurrentVirtualTime(origin + milliseconds(scenario.ballotMs)); - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); clock.setCurrentVirtualTime(origin + milliseconds(scenario.closeMs)); clock.setSystemTimeOffset(milliseconds(scenario.clockOffsetMs)); - REQUIRE(driver.getNominationTimeouts(slot).value() == 0); + REQUIRE(driver.getLeaderTimeouts(slot).value() == 0); REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(scenario.recordTrigger ? scenario.ballotMs : 0)); REQUIRE(driver.getTriggerToBallotDuration(slot + 1) == @@ -6926,7 +5536,7 @@ TEST_CASE("trigger fallback credits elapsed time through nomination", auto const before = fallback.count(); auto deadline = std::max( clock.now(), - EarlyNominationTestAccess::triggerAnchor(herder, slot) + seconds(2)); + LeaderBuildTestAccess::triggerAnchor(herder, slot) + seconds(2)); REQUIRE(duration_cast(deadline - clock.now()).count() == scenario.expectedWaitMs); REQUIRE(fallback.count() - before == scenario.expectedFallbacks); @@ -6946,7 +5556,7 @@ TEST_CASE("trigger work is recorded by the proposal path", SecretKey::pseudoRandomForTesting().getPublicKey()}; auto app = createTestApplication(clock, cfg); auto& herder = static_cast(app->getHerder()); - EarlyNominationTestAccess::selectLocalFirstLeader(herder); + LeaderBuildTestAccess::selectLocalFirstLeader(herder); auto& driver = herder.getHerderSCPDriver(); auto const slot = app->getLedgerManager().getLastClosedLedgerNum() + 1; herder.mGetTopTransactionsForTesting = [&](size_t) { @@ -6957,19 +5567,16 @@ TEST_CASE("trigger work is recorded by the proposal path", return std::vector{}; }; herder.triggerNextLedger(slot, true); - // Until ballot starts there is no completed interval to credit. - REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds::zero()); - clock.setCurrentVirtualTime(clock.now() + milliseconds(100)); - driver.recordSCPEvent(slot, false); - REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(500)); + // The round leader proposes straight into the ballot protocol: the + // interval ends when its own build does. + REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(400)); clock.setCurrentVirtualTime(clock.now() + seconds(1)); - driver.recordNominationTrigger(slot); - driver.recordSCPEvent(slot, true); - REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(500)); + driver.recordTrigger(slot); + REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(400)); herder.mGetTopTransactionsForTesting = nullptr; } -TEST_CASE("skipped nomination does not start the trigger interval", +TEST_CASE("skipped trigger does not start the trigger interval", "[herder][trigger-construction]") { using namespace std::chrono; @@ -6977,7 +5584,7 @@ TEST_CASE("skipped nomination does not start the trigger interval", auto cfg = getTestConfig(); cfg.HTTP_PORT = 0; cfg.MANUAL_CLOSE = true; - // A second validator keeps our own nomination from externalizing. + // A second validator keeps our own proposal from externalizing. cfg.QUORUM_SET.threshold = 2; cfg.QUORUM_SET.validators = { cfg.NODE_SEED.getPublicKey(), @@ -6996,7 +5603,7 @@ TEST_CASE("skipped nomination does not start the trigger interval", clock.setSystemTimeOffset(-(Herder::MAX_TIME_SLIP_SECONDS + seconds(10))); // Counts proposal constructions, which happen only for triggers that go - // on to nominate. + // on to start the slot. int builds = 0; herder.mGetTopTransactionsForTesting = [&](size_t) { ++builds; @@ -7009,7 +5616,7 @@ TEST_CASE("skipped nomination does not start the trigger interval", // Peers move the slot into ballot while we wait. None of that waiting is // trigger-to-ballot work. clock.setCurrentVirtualTime(clock.now() + milliseconds(700)); - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds::zero()); // With the clock back in sync, the same path builds a proposal and its @@ -7020,7 +5627,7 @@ TEST_CASE("skipped nomination does not start the trigger interval", herder.triggerNextLedger(nextSlot, true); REQUIRE(builds == 1); clock.setCurrentVirtualTime(clock.now() + milliseconds(300)); - driver.recordSCPEvent(nextSlot, false); + driver.recordBallotStart(nextSlot); REQUIRE(driver.getTriggerToBallotDuration(nextSlot) == milliseconds(300)); herder.mGetTopTransactionsForTesting = nullptr; } @@ -7037,16 +5644,14 @@ TEST_CASE("trigger work excludes time after entering ballot", auto& driver = static_cast(app->getHerder()).getHerderSCPDriver(); uint64_t const slot = 100; - driver.recordNominationTrigger(slot); + driver.recordTrigger(slot); clock.setCurrentVirtualTime(clock.now() + milliseconds(100)); - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); clock.setCurrentVirtualTime(clock.now() + milliseconds(300)); - driver.recordSCPEvent(slot, true); REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(100)); - driver.recordSCPEvent(slot + 1, false); + driver.recordBallotStart(slot + 1); clock.setCurrentVirtualTime(clock.now() + milliseconds(100)); - driver.recordNominationTrigger(slot + 1); - driver.recordSCPEvent(slot + 1, true); + driver.recordTrigger(slot + 1); REQUIRE(driver.getTriggerToBallotDuration(slot + 1) == milliseconds::zero()); } @@ -7067,19 +5672,17 @@ TEST_CASE("trigger allowance is independent of timeout callback ordering", auto const ct = VirtualClock::to_time_t(clock.system_now()) + 1; auto const origin = VirtualClock::from_time_t(ct); clock.setCurrentVirtualTime(origin); - driver.recordNominationTrigger(slot); + driver.recordTrigger(slot); clock.setCurrentVirtualTime(origin + milliseconds(100)); - driver.recordSCPEvent(slot, true); bool timerFired = false; - driver.setupTimer(slot, Slot::NOMINATION_TIMER, seconds(2), + driver.setupTimer(slot, Slot::LEADER_TIMER, seconds(2), [&] { timerFired = true; }); - // Both events become runnable while the main thread is occupied. Entering - // ballot before dispatching the expired timer suppresses its timeout count. + // Both events become runnable while the main thread is occupied. clock.setCurrentVirtualTime(origin + milliseconds(2500)); if (ballotBeforeTimeout) { - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); } for (int i = 0; i < 100 && !timerFired; ++i) { @@ -7089,18 +5692,16 @@ TEST_CASE("trigger allowance is independent of timeout callback ordering", REQUIRE(clock.system_now() == origin + milliseconds(2500)); if (!ballotBeforeTimeout) { - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); } - REQUIRE(driver.getNominationTimeouts(slot) == - (ballotBeforeTimeout ? 0 : 1)); + REQUIRE(driver.getLeaderTimeouts(slot) == 1); REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(2500)); StellarValue value; value.closeTime = ct; herder.setTrackingSCPState(slot, value, true); clock.setCurrentVirtualTime(origin + milliseconds(2700)); - REQUIRE(EarlyNominationTestAccess::triggerAnchor(herder, slot) + - seconds(2) <= + REQUIRE(LeaderBuildTestAccess::triggerAnchor(herder, slot) + seconds(2) <= clock.now()); } @@ -7133,10 +5734,9 @@ TEST_CASE("trigger anchor keeps millisecond close times", // 600 ms later, so the allowance is exactly 600 ms. auto const origin = getConsensusTime(value).toSystemTime(); clock.setCurrentVirtualTime(origin); - driver.recordNominationTrigger(slot); - driver.recordSCPEvent(slot, true); + driver.recordTrigger(slot); clock.setCurrentVirtualTime(origin + milliseconds(600)); - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(600)); auto metrics = app->getMetrics().GetAllMetrics(); @@ -7145,10 +5745,9 @@ TEST_CASE("trigger anchor keeps millisecond close times", auto const before = fallback.count(); auto waitAt = [&](milliseconds sinceClose) { clock.setCurrentVirtualTime(origin + sinceClose); - auto deadline = - std::max(clock.now(), - EarlyNominationTestAccess::triggerAnchor(herder, slot) + - seconds(2)); + auto deadline = std::max( + clock.now(), + LeaderBuildTestAccess::triggerAnchor(herder, slot) + seconds(2)); return duration_cast(deadline - clock.now()); }; @@ -7158,7 +5757,8 @@ TEST_CASE("trigger anchor keeps millisecond close times", REQUIRE(fallback.count() == before); // Elapsed time up to the target plus the allowance is still explained by - // nomination; one millisecond more is attributed to clock drift. + // the wait for a proposal; one millisecond more is attributed to clock + // drift. REQUIRE(waitAt(milliseconds(2600)) == milliseconds::zero()); REQUIRE(fallback.count() == before); REQUIRE(waitAt(milliseconds(2601)) == milliseconds::zero()); @@ -7180,27 +5780,31 @@ TEST_CASE("trigger-to-ballot time is recorded at first ballot entry", app->getMetrics().NewTimer({"scp", "timing", "trigger-to-ballot"}); REQUIRE(timer.count() == 0); - // Construction and nomination both count, up to the first ballot entry. + // Construction and the wait for the leader both count, up to the first + // ballot entry. uint64_t const slot = 100; - driver.recordNominationTrigger(slot); + driver.recordTrigger(slot); clock.setCurrentVirtualTime(clock.now() + milliseconds(400)); - driver.recordSCPEvent(slot, true); clock.setCurrentVirtualTime(clock.now() + milliseconds(100)); - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); REQUIRE(timer.count() == 1); REQUIRE(timer.max() == Approx(500.0)); // Later ballot events of the same slot are not new intervals. clock.setCurrentVirtualTime(clock.now() + milliseconds(50)); - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); REQUIRE(timer.count() == 1); - // No interval without a local trigger, or when ballot started first. - driver.recordSCPEvent(slot + 1, false); - driver.recordSCPEvent(slot + 2, false); - driver.recordNominationTrigger(slot + 2); - driver.recordSCPEvent(slot + 2, true); + // No interval without a local trigger. + driver.recordBallotStart(slot + 1); REQUIRE(timer.count() == 1); + + // A trigger that fires after the node already voted for the round + // leader's proposal records a zero interval. + driver.recordBallotStart(slot + 2); + driver.recordTrigger(slot + 2); + REQUIRE(timer.count() == 2); + REQUIRE(timer.min() == Approx(0.0)); } TEST_CASE("trigger lateness and immediate triggers are recorded", @@ -7255,7 +5859,7 @@ TEST_CASE("trigger lateness and immediate triggers are recorded", clock.setCurrentVirtualTime(std::max(overdue, clock.now()) + milliseconds(500)); auto const immediateCount = immediate.count(); - EarlyNominationTestAccess::setupTrigger(herder); + LeaderBuildTestAccess::setupTrigger(herder); REQUIRE(immediate.count() == immediateCount + 1); REQUIRE(herder.getTriggerTimer().expiry_time() == clock.now()); testutil::crankUntil( @@ -7263,7 +5867,7 @@ TEST_CASE("trigger lateness and immediate triggers are recorded", REQUIRE(lm.getLastClosedLedgerNum() > lcl); } -TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") +TEST_CASE("prepare proposal before trigger", "[herder][early-proposal]") { auto const soroban = GENERATE(false, true); VirtualClock clock; @@ -7276,7 +5880,7 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") cfg.GENESIS_TEST_ACCOUNT_COUNT = 10; cfg.TESTING_UPGRADE_MAX_TX_SET_SIZE = 1; // Require another node to externalize, so this test can inspect the value - // we nominate without immediately closing it and scheduling another slot. + // we propose without immediately closing it and scheduling another slot. cfg.QUORUM_SET.threshold = 2; cfg.QUORUM_SET.validators = { cfg.NODE_SEED.getPublicKey(), @@ -7289,7 +5893,7 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") config.mLedgerMaxTxCount = 1; }); } - EarlyNominationTestAccess::selectLocalFirstLeader(herder); + LeaderBuildTestAccess::selectLocalFirstLeader(herder); auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); // The Soroban upgrade helper closes ledgers at its fixed test date. The // sub-second offset makes the prepared close time carry milliseconds once @@ -7298,7 +5902,7 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") VirtualClock::from_time_t(lcl.header.scpValue.closeTime + 1000) + std::chrono::milliseconds(375)); auto const seq = lcl.header.ledgerSeq + 1; - herder.getHerderSCPDriver().recordSCPEvent(lcl.header.ledgerSeq, false); + herder.getHerderSCPDriver().recordBallotStart(lcl.header.ledgerSeq); std::vector mempool; auto alice = txtest::getGenesisAccount(*app, 0); @@ -7339,18 +5943,16 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") REQUIRE(trigger > clock.now()); testutil::crankUntil( app, - [&]() { - return EarlyNominationTestAccess::prepared(herder) != nullptr; - }, + [&]() { return LeaderBuildTestAccess::prepared(herder) != nullptr; }, std::chrono::seconds(1)); REQUIRE(clock.now() < trigger); REQUIRE(pulls == 1); - auto prepared = EarlyNominationTestAccess::prepared(herder); - auto const preparedCloseTime = EarlyNominationTestAccess::closeTime(herder); + auto prepared = LeaderBuildTestAccess::prepared(herder); + auto const preparedCloseTime = LeaderBuildTestAccess::closeTime(herder); REQUIRE(prepared); REQUIRE(prepared->sizeTxTotal() == 1); REQUIRE(preparedCloseTime == expectedConsensusTime); - REQUIRE(herder.getSCP().getNominationLeaders(seq).empty()); + REQUIRE(herder.getSCP().getLeaderRound(seq) == 0); REQUIRE(herder.getSCP().getLatestMessagesSend(seq).empty()); auto known = herder.getTxSet(prepared->getContentsHash()); REQUIRE(std::get(known) == nullptr); @@ -7361,13 +5963,10 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") // used to validate time bounds, not acquire the new wall-clock time. clock.setCurrentVirtualTime(trigger + std::chrono::seconds(1)); testutil::crankUntil( - app, - [&]() { - return !herder.getSCP().getNominationLeaders(seq).empty(); - }, + app, [&]() { return herder.getSCP().getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); REQUIRE(pulls == 1); - REQUIRE(!EarlyNominationTestAccess::prepared(herder)); + REQUIRE(!LeaderBuildTestAccess::prepared(herder)); REQUIRE(std::get( herder.getTxSet(prepared->getContentsHash())) == prepared); // The value can be checked independently of the production cache. @@ -7387,10 +5986,12 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") std::chrono::seconds(1)); auto messages = herder.getSCP().getLatestMessagesSend(seq); REQUIRE(!messages.empty()); - auto const& votes = messages.front().statement.pledges.nominate().votes; - REQUIRE(votes.size() == 1); + REQUIRE(messages.front().statement.pledges.type() == SCP_ST_PREPARE); + auto const& ballot = + messages.front().statement.pledges.prepare().ballot; + REQUIRE(ballot.counter == 1); StellarValue value; - xdr::xdr_from_opaque(votes.front(), value); + xdr::xdr_from_opaque(ballot.value, value); REQUIRE(value.txSetHash == prepared->getContentsHash()); REQUIRE(getConsensusTime(value) == preparedCloseTime); REQUIRE(pulls == 1); @@ -7402,7 +6003,7 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") // claim that our time-bound transaction is valid for that time. herder.triggerNextLedger(seq, true); REQUIRE(pulls == 2); - REQUIRE(!EarlyNominationTestAccess::prepared(herder)); + REQUIRE(!LeaderBuildTestAccess::prepared(herder)); REQUIRE(std::get( herder.getTxSet(prepared->getContentsHash())) == nullptr); } @@ -7410,32 +6011,32 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") { herder.externalizeValue(TxSetXDRFrame::makeEmpty(lcl), seq, preparedCloseTime, {}, cfg.NODE_SEED); - REQUIRE(EarlyNominationTestAccess::prepared(herder) != prepared); + REQUIRE(LeaderBuildTestAccess::prepared(herder) != prepared); auto const pullsAfterClose = pulls; - // A stale timer/caller cannot nominate the old slot or pull again. + // A stale timer/caller cannot propose for the old slot or pull again. herder.triggerNextLedger(seq, true); REQUIRE(pulls == pullsAfterClose); } SECTION("loss of sync discards preparation") { herder.lostSync(); - REQUIRE(!EarlyNominationTestAccess::prepared(herder)); + REQUIRE(!LeaderBuildTestAccess::prepared(herder)); herder.triggerNextLedger(seq, true); REQUIRE(pulls == 1); } SECTION("shutdown cancels preparation") { herder.shutdown(); - REQUIRE(!EarlyNominationTestAccess::prepared(herder)); + REQUIRE(!LeaderBuildTestAccess::prepared(herder)); } herder.mGetTopTransactionsForTesting = nullptr; } -TEST_CASE("only nomination leaders construct local proposals", - "[herder][early-nomination]") +TEST_CASE("only round leaders construct local proposals", + "[herder][early-proposal]") { - // A fixed, unanimous quorum gives all three nodes the same weights and - // leader ordering. Exercise each member as the local node, rather than + // A fixed, unanimous quorum gives all three nodes the same leader + // schedule. Exercise each member as the local node, rather than // hoping random peers cover every role for one fixed local priority. std::vector keys; SCPQuorumSet qset; @@ -7443,14 +6044,14 @@ TEST_CASE("only nomination leaders construct local proposals", for (int i = 0; i < 3; ++i) { keys.push_back(SecretKey::fromSeed( - sha256(fmt::format("early-nomination-node-{}", i)))); + sha256(fmt::format("early-proposal-node-{}", i)))); qset.validators.push_back(keys.back().getPublicKey()); } std::set checkedRoles; - std::optional previous; - std::optional> expectedFirst; - std::optional> expectedFirstTwo; + std::optional previous; + std::optional expectedFirst; + std::optional expectedSecond; for (int i = 0; i < 3; ++i) { VirtualClock clock; @@ -7467,36 +6068,38 @@ TEST_CASE("only nomination leaders construct local proposals", auto& scp = herder.getSCP(); auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); auto const seq = lcl.header.ledgerSeq + 1; - auto const value = xdr::xdr_to_opaque(lcl.header.scpValue); + // The schedule is seeded by the hash of the last closed ledger. if (!previous) { - previous = value; + previous = lcl.hash; } - REQUIRE(value == *previous); - herder.getHerderSCPDriver().recordSCPEvent(lcl.header.ledgerSeq, false); + REQUIRE(lcl.hash == *previous); + auto& driver = herder.getHerderSCPDriver(); + herder.getHerderSCPDriver().recordBallotStart(lcl.header.ledgerSeq); size_t pulls = 0; herder.mGetTopTransactionsForTesting = [&](size_t) { ++pulls; return std::vector{}; }; - auto first = scp.predictNominationLeaders(seq, value, 1); - auto firstTwo = scp.predictNominationLeaders(seq, value, 2); - REQUIRE(first.size() == 1); - REQUIRE(firstTwo.size() == 2); + auto first = driver.getRoundLeader(seq, 1); + auto second = driver.getRoundLeader(seq, 2); + REQUIRE(first); + REQUIRE(second); + REQUIRE(*first != *second); if (!expectedFirst) { expectedFirst = first; - expectedFirstTwo = firstTwo; + expectedSecond = second; } - REQUIRE(first == *expectedFirst); - REQUIRE(firstTwo == *expectedFirstTwo); - int role = first.count(scp.getLocalNodeID()) ? 0 - : firstTwo.count(scp.getLocalNodeID()) ? 1 - : 2; + REQUIRE(*first == *expectedFirst); + REQUIRE(*second == *expectedSecond); + int role = *first == scp.getLocalNodeID() ? 0 + : *second == scp.getLocalNodeID() ? 1 + : 2; REQUIRE(checkedRoles.insert(role).second); CAPTURE(role); - EarlyNominationTestAccess::schedule(herder); - REQUIRE(EarlyNominationTestAccess::scheduled(herder) == (role == 0)); + LeaderBuildTestAccess::schedule(herder); + REQUIRE(LeaderBuildTestAccess::scheduled(herder) == (role == 0)); bool done = false; VirtualTimer stop(clock); stop.expires_from_now(std::chrono::milliseconds(1)); @@ -7506,42 +6109,42 @@ TEST_CASE("only nomination leaders construct local proposals", REQUIRE(done); REQUIRE(clock.now() < herder.getTriggerTimer().expiry_time()); REQUIRE(pulls == (role == 0 ? 1 : 0)); - REQUIRE(bool(EarlyNominationTestAccess::prepared(herder)) == - (role == 0)); - REQUIRE(scp.getNominationLeaders(seq).empty()); + REQUIRE(bool(LeaderBuildTestAccess::prepared(herder)) == (role == 0)); + REQUIRE(scp.getLeaderRound(seq) == 0); REQUIRE(scp.getLatestMessagesSend(seq).empty()); auto const trigger = herder.getTriggerTimer().expiry_time(); clock.setCurrentVirtualTime(trigger); auto const triggerSystemTime = clock.system_now(); testutil::crankUntil( - app, [&]() { return !scp.getNominationLeaders(seq).empty(); }, + app, [&]() { return scp.getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); - REQUIRE(scp.getNominationLeaders(seq) == first); + REQUIRE(scp.getLeaderRound(seq) == 1); // The first leader refreshes its empty early snapshot. Followers // pull the mempool once, only to pre-verify signatures; they do not // build or publish a set. REQUIRE(pulls == (role == 0 ? 2 : 1)); REQUIRE(scp.getLatestMessagesSend(seq).empty() == (role != 0)); - // With the first leader silent, real nomination timers eventually + // With the first leader silent, real leader timers eventually // promote each follower. It builds exactly then, using a current close // time, and its prepared/validated tx set is available for retrieval. testutil::crankUntil( app, [&]() { return !scp.getLatestMessagesSend(seq).empty(); }, std::chrono::seconds(30)); - REQUIRE(scp.getNominationLeaders(seq).count(scp.getLocalNodeID())); + REQUIRE(scp.getLeaderRound(seq) == static_cast(role + 1)); + REQUIRE(driver.isLocalNodeRoundLeader(seq, scp.getLeaderRound(seq))); REQUIRE(pulls == 2); auto messages = scp.getLatestMessagesSend(seq); - auto const& votes = messages.front().statement.pledges.nominate().votes; - REQUIRE(votes.size() == 1); + REQUIRE(messages.front().statement.pledges.type() == SCP_ST_PREPARE); + auto const& proposed = + messages.front().statement.pledges.prepare().ballot.value; StellarValue nominated; - xdr::xdr_from_opaque(votes.front(), nominated); + xdr::xdr_from_opaque(proposed, nominated); auto txSet = std::get( herder.getTxSet(nominated.txSetHash)); REQUIRE(txSet); REQUIRE(txSet->previousLedgerHash() == lcl.hash); - REQUIRE(herder.getHerderSCPDriver().validateValue(seq, votes.front(), - true) == + REQUIRE(driver.validateValue(seq, proposed, true) == SCPDriver::kFullyValidatedValue); if (role != 0) { @@ -7559,10 +6162,10 @@ TEST_CASE("only nomination leaders construct local proposals", } TEST_CASE("a follower promoted after losing sync still proposes", - "[herder][early-nomination]") + "[herder][early-proposal]") { // Unanimous three-node quorum with silent peers: the slot cannot - // externalize, so the local follower eventually loses sync. Nomination + // externalize, so the local follower eventually loses sync. Leader // timeouts must still let it propose a value of its own once promoted. std::vector keys; SCPQuorumSet qset; @@ -7570,7 +6173,7 @@ TEST_CASE("a follower promoted after losing sync still proposes", for (int i = 0; i < 3; ++i) { keys.push_back(SecretKey::fromSeed( - sha256(fmt::format("early-nomination-node-{}", i)))); + sha256(fmt::format("early-proposal-node-{}", i)))); qset.validators.push_back(keys.back().getPublicKey()); } @@ -7591,27 +6194,26 @@ TEST_CASE("a follower promoted after losing sync still proposes", auto& scp = herder.getSCP(); auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); auto const seq = lcl.header.ledgerSeq + 1; - if (scp.predictNominationLeaders( - seq, xdr::xdr_to_opaque(lcl.header.scpValue), 1) - .count(scp.getLocalNodeID())) + auto& driver = herder.getHerderSCPDriver(); + if (driver.isLocalNodeRoundLeader(seq, 1)) { // The first leader proposes at the trigger; test a follower. continue; } checkedFollower = true; - herder.getHerderSCPDriver().recordSCPEvent(lcl.header.ledgerSeq, false); + herder.getHerderSCPDriver().recordBallotStart(lcl.header.ledgerSeq); size_t pulls = 0; herder.mGetTopTransactionsForTesting = [&](size_t) { ++pulls; return std::vector{}; }; - EarlyNominationTestAccess::schedule(herder); + LeaderBuildTestAccess::schedule(herder); clock.setCurrentVirtualTime(herder.getTriggerTimer().expiry_time()); testutil::crankUntil( - app, [&]() { return !scp.getNominationLeaders(seq).empty(); }, + app, [&]() { return scp.getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); - REQUIRE(!scp.getNominationLeaders(seq).empty()); + REQUIRE(scp.getLeaderRound(seq) != 0); REQUIRE(scp.getLatestMessagesSend(seq).empty()); // Only the signature prewarm pull; no build yet. REQUIRE(pulls == 1); @@ -7623,29 +6225,29 @@ TEST_CASE("a follower promoted after losing sync still proposes", testutil::crankUntil( app, [&]() { return !scp.getLatestMessagesSend(seq).empty(); }, std::chrono::seconds(30)); - REQUIRE(scp.getNominationLeaders(seq).count(scp.getLocalNodeID())); + REQUIRE(driver.isLocalNodeRoundLeader(seq, scp.getLeaderRound(seq))); REQUIRE(pulls == 2); auto messages = scp.getLatestMessagesSend(seq); REQUIRE(!messages.empty()); - auto const& votes = messages.front().statement.pledges.nominate().votes; - REQUIRE(votes.size() == 1); - REQUIRE(herder.getHerderSCPDriver().validateValue(seq, votes.front(), - true) == - SCPDriver::kFullyValidatedValue); + REQUIRE(messages.front().statement.pledges.type() == SCP_ST_PREPARE); + REQUIRE(driver.validateValue( + seq, + messages.front().statement.pledges.prepare().ballot.value, + true) == SCPDriver::kFullyValidatedValue); herder.mGetTopTransactionsForTesting = nullptr; } REQUIRE(checkedFollower); } TEST_CASE("a leader build that completes a fetch can externalize", - "[herder][early-nomination]") + "[herder][early-proposal]") { // The local leader builds the very set a peer's ballot is waiting for (an // empty set is the same on every node). Publishing it completes the fetch - // inside SCP's nomination value supplier: the peer's CONFIRM is + // inside SCP's proposal value supplier: the peer's CONFIRM is // reprocessed there and, the peer being v-blocking, externalizes the slot - // before the supplier returns. Nomination must then stop cleanly without - // voting or emitting. + // before the supplier returns. The leader must then stop cleanly without + // proposing its own value. VirtualClock clock; clock.setCurrentVirtualTime(VirtualClock::from_time_t(1000)); auto cfg = getTestConfig(); @@ -7661,7 +6263,7 @@ TEST_CASE("a leader build that completes a fetch can externalize", auto app = createTestApplication(clock, cfg); auto& herder = static_cast(app->getHerder()); auto& scp = herder.getSCP(); - EarlyNominationTestAccess::selectLocalFirstLeader(herder); + LeaderBuildTestAccess::selectLocalFirstLeader(herder); auto const qset = scp.getLocalQuorumSet(); auto const qsetHash = scp.getLocalNode()->getQuorumSetHash(); std::optional peer; @@ -7679,7 +6281,7 @@ TEST_CASE("a leader build that completes a fetch can externalize", auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); auto const seq = lcl.header.ledgerSeq + 1; - herder.getHerderSCPDriver().recordSCPEvent(lcl.header.ledgerSeq, false); + herder.getHerderSCPDriver().recordBallotStart(lcl.header.ledgerSeq); herder.mGetTopTransactionsForTesting = [](size_t) { return std::vector{}; }; @@ -7725,13 +6327,13 @@ TEST_CASE("a leader build that completes a fetch can externalize", REQUIRE(closed.header.scpValue == peerValue); for (auto const& env : scp.getLatestMessagesSend(seq)) { - REQUIRE(env.statement.pledges.type() != SCP_ST_NOMINATE); + REQUIRE(env.statement.pledges.type() == SCP_ST_EXTERNALIZE); } herder.mGetTopTransactionsForTesting = nullptr; } TEST_CASE("followers pre-verify mempool signatures at the trigger", - "[herder][early-nomination]") + "[herder][early-proposal]") { // Followers no longer validate their own mempool by building a set, so // the leader's set would be validated with a cold signature cache. At the @@ -7742,7 +6344,7 @@ TEST_CASE("followers pre-verify mempool signatures at the trigger", for (int i = 0; i < 3; ++i) { keys.push_back(SecretKey::fromSeed( - sha256(fmt::format("early-nomination-node-{}", i)))); + sha256(fmt::format("early-proposal-node-{}", i)))); qset.validators.push_back(keys.back().getPublicKey()); } @@ -7765,9 +6367,7 @@ TEST_CASE("followers pre-verify mempool signatures at the trigger", auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); auto const seq = lcl.header.ledgerSeq + 1; bool const leader = - scp.predictNominationLeaders( - seq, xdr::xdr_to_opaque(lcl.header.scpValue), 1) - .count(scp.getLocalNodeID()) != 0; + herder.getHerderSCPDriver().isLocalNodeRoundLeader(seq, 1); if (!checkedRoles.insert(leader).second) { continue; @@ -7797,7 +6397,7 @@ TEST_CASE("followers pre-verify mempool signatures at the trigger", { envelopes.push_back(tx->getEnvelope()); } - herder.getHerderSCPDriver().recordSCPEvent(lcl.header.ledgerSeq, false); + herder.getHerderSCPDriver().recordBallotStart(lcl.header.ledgerSeq); herder.mGetTopTransactionsForTesting = [&](size_t) { return envelopes; }; @@ -7807,12 +6407,12 @@ TEST_CASE("followers pre-verify mempool signatures at the trigger", PubKeyUtils::flushVerifySigCacheCounts(hits, misses); size_t const signatures = txs.size() + 1 /* op source */ + 1 /* fee bump's inner tx */; - EarlyNominationTestAccess::schedule(herder); + LeaderBuildTestAccess::schedule(herder); clock.setCurrentVirtualTime(herder.getTriggerTimer().expiry_time()); testutil::crankUntil( - app, [&]() { return !scp.getNominationLeaders(seq).empty(); }, + app, [&]() { return scp.getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); - REQUIRE(!scp.getNominationLeaders(seq).empty()); + REQUIRE(scp.getLeaderRound(seq) != 0); if (leader) { @@ -7825,7 +6425,7 @@ TEST_CASE("followers pre-verify mempool signatures at the trigger", } // Background tasks verify every signature once. Wait in real time - // without cranking: virtual nomination timeouts would promote this + // without cranking: virtual leader timeouts would promote this // follower, whose build verifies signatures too. uint64_t verified = 0; auto const deadline = @@ -7929,25 +6529,24 @@ TEST_CASE("signature prewarm at the ballot covers the next ledger", ballot.counter = 1; ballot.value = value; herder.getHerderSCPDriver().startedBallotProtocol(slot, ballot); - REQUIRE(EarlyNominationTestAccess::prewarmLedger(herder) == 0); + REQUIRE(LeaderBuildTestAccess::prewarmLedger(herder) == 0); testutil::crankUntil( app, [&]() { - return EarlyNominationTestAccess::prewarmLedger(herder) == slot + 1; + return LeaderBuildTestAccess::prewarmLedger(herder) == slot + 1; }, std::chrono::seconds(1)); - REQUIRE(EarlyNominationTestAccess::prewarmLedger(herder) == slot + 1); + REQUIRE(LeaderBuildTestAccess::prewarmLedger(herder) == slot + 1); REQUIRE(requested == - 2 * EarlyNominationTestAccess::maxTxSetCandidates(herder) + - valueTxs); - REQUIRE(EarlyNominationTestAccess::waitForPrewarm(herder)); + 2 * LeaderBuildTestAccess::maxTxSetCandidates(herder) + valueTxs); + REQUIRE(LeaderBuildTestAccess::waitForPrewarm(herder)); // N externalizes and closes; the prewarm for N + 1 is kept. The overlay // drops N's transactions from the mempool. herder.externalizeValue(valueSet, slot, closeTime, emptyUpgradeSteps, cfg.NODE_SEED); REQUIRE(lm.getLastClosedLedgerNum() == slot); - REQUIRE(EarlyNominationTestAccess::prewarmLedger(herder) == slot + 1); + REQUIRE(LeaderBuildTestAccess::prewarmLedger(herder) == slot + 1); mempool.erase(mempool.begin(), mempool.begin() + valueTxs); auto const nextLcl = lm.getLastClosedLedgerHeader(); auto const nextCloseTime = makeConsensusTime(app->timeNow() + 2); @@ -7979,7 +6578,7 @@ TEST_CASE("signature prewarm at the ballot covers the next ledger", { auto [hits0, misses0] = lookups("build"); auto txSet = - EarlyNominationTestAccess::build(herder, slot + 1, nextCloseTime); + LeaderBuildTestAccess::build(herder, slot + 1, nextCloseTime); REQUIRE(txSet->sizeTxTotal() == 16); checkLookups("build", hits0, misses0, target); } @@ -8005,7 +6604,7 @@ TEST_CASE("signature prewarm at the ballot covers the next ledger", } // Validation and the build stop the prewarm and record what it did: the // 16 transactions of N's value and the next 20, each verified once. - REQUIRE(EarlyNominationTestAccess::prewarmLedger(herder) == 0); + REQUIRE(LeaderBuildTestAccess::prewarmLedger(herder) == 0); REQUIRE(prewarmTxs.count() == txs0 + valueTxs + target); REQUIRE(prewarmVerified.count() == verified0 + valueTxs + target); REQUIRE(prewarmIncomplete.count() == incomplete0); @@ -8013,7 +6612,7 @@ TEST_CASE("signature prewarm at the ballot covers the next ledger", } TEST_CASE("early underfilled proposals refresh at the trigger", - "[herder][early-nomination]") + "[herder][early-proposal]") { auto const invalidFirst = GENERATE(false, true); VirtualClock clock; @@ -8030,10 +6629,10 @@ TEST_CASE("early underfilled proposals refresh at the trigger", SecretKey::pseudoRandomForTesting().getPublicKey()}; auto app = createTestApplication(clock, cfg); auto& herder = static_cast(app->getHerder()); - EarlyNominationTestAccess::selectLocalFirstLeader(herder); + LeaderBuildTestAccess::selectLocalFirstLeader(herder); auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); auto const seq = lcl.header.ledgerSeq + 1; - herder.getHerderSCPDriver().recordSCPEvent(lcl.header.ledgerSeq, false); + herder.getHerderSCPDriver().recordBallotStart(lcl.header.ledgerSeq); auto alice = txtest::getGenesisAccount(*app, 0); auto bob = txtest::getGenesisAccount(*app, 1); auto tx = alice.tx({payment(bob, 100)}); @@ -8051,24 +6650,21 @@ TEST_CASE("early underfilled proposals refresh at the trigger", ++pulls; return mempool; }; - EarlyNominationTestAccess::schedule(herder); + LeaderBuildTestAccess::schedule(herder); testutil::crankUntil( app, - [&]() { - return EarlyNominationTestAccess::prepared(herder) != nullptr; - }, + [&]() { return LeaderBuildTestAccess::prepared(herder) != nullptr; }, std::chrono::seconds(1)); REQUIRE(pulls == 1); - REQUIRE(EarlyNominationTestAccess::prepared(herder)->sizeTxTotal() == 0); + REQUIRE(LeaderBuildTestAccess::prepared(herder)->sizeTxTotal() == 0); // Transactions arrive while we are waiting for the normal trigger. mempool = {tx->getEnvelope(), extra->getEnvelope()}; clock.setCurrentVirtualTime(herder.getTriggerTimer().expiry_time()); testutil::crankUntil( - app, - [&]() { return !herder.getSCP().getNominationLeaders(seq).empty(); }, + app, [&]() { return herder.getSCP().getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); REQUIRE(pulls == 2); - // Check the actual set cached for nomination, independently of benchmark + // Check the actual set cached for the proposal, independently of benchmark // instrumentation. Both newly arrived transactions must be present. auto expected = makeTxSetFromTransactions(TxFrameList{tx, extra}, *app, ApplyTimeOffset{}) @@ -8078,12 +6674,12 @@ TEST_CASE("early underfilled proposals refresh at the trigger", herder.getTxSet(expected->getContentsHash())); REQUIRE(nominated); REQUIRE(nominated->sizeTxTotal() == 2); - REQUIRE(!EarlyNominationTestAccess::prepared(herder)); + REQUIRE(!LeaderBuildTestAccess::prepared(herder)); herder.mGetTopTransactionsForTesting = nullptr; } TEST_CASE("early preparation respects manual and immediately due triggers", - "[herder][early-nomination]") + "[herder][early-proposal]") { auto const manual = GENERATE(false, true); VirtualClock clock; @@ -8098,7 +6694,7 @@ TEST_CASE("early preparation respects manual and immediately due triggers", SecretKey::pseudoRandomForTesting().getPublicKey()}; auto app = createTestApplication(clock, cfg); auto& herder = static_cast(app->getHerder()); - EarlyNominationTestAccess::selectLocalFirstLeader(herder); + LeaderBuildTestAccess::selectLocalFirstLeader(herder); auto const seq = app->getLedgerManager().getLastClosedLedgerNum() + 1; size_t pulls = 0; herder.mGetTopTransactionsForTesting = [&](size_t) { @@ -8107,29 +6703,26 @@ TEST_CASE("early preparation respects manual and immediately due triggers", }; if (manual) { - REQUIRE(!EarlyNominationTestAccess::scheduled(herder)); + REQUIRE(!LeaderBuildTestAccess::scheduled(herder)); REQUIRE(herder.getTriggerTimer().seq() == 0); herder.triggerNextLedger(seq, true); } else { - EarlyNominationTestAccess::schedule(herder); + LeaderBuildTestAccess::schedule(herder); // No previous prepare timestamp: the normal fallback triggers now. REQUIRE(herder.getTriggerTimer().expiry_time() == clock.now()); // A due trigger builds itself. Preparing too could build twice: the // two timers would fire at the same instant in either order. - REQUIRE(!EarlyNominationTestAccess::scheduled(herder)); + REQUIRE(!LeaderBuildTestAccess::scheduled(herder)); testutil::crankUntil( - app, - [&]() { - return !herder.getSCP().getNominationLeaders(seq).empty(); - }, + app, [&]() { return herder.getSCP().getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); } REQUIRE(pulls == 1); - REQUIRE(!EarlyNominationTestAccess::prepared(herder)); - REQUIRE(!EarlyNominationTestAccess::scheduled(herder)); - REQUIRE(!herder.getSCP().getNominationLeaders(seq).empty()); + REQUIRE(!LeaderBuildTestAccess::prepared(herder)); + REQUIRE(!LeaderBuildTestAccess::scheduled(herder)); + REQUIRE(herder.getSCP().getLeaderRound(seq) != 0); herder.mGetTopTransactionsForTesting = nullptr; } @@ -8553,7 +7146,7 @@ TEST_CASE("leader builds validate only the top of the mempool pull", }; auto const lcl = lm.getLastClosedLedgerHeader(); - auto [txSet, invalid] = EarlyNominationTestAccess::buildWithInvalid( + auto [txSet, invalid] = LeaderBuildTestAccess::buildWithInvalid( herder, lcl.header.ledgerSeq + 1, getConsensusTime(lcl.header.scpValue).next(lcl.header.ledgerVersion)); herder.mGetTopTransactionsForTesting = nullptr; @@ -8693,8 +7286,7 @@ TEST_CASE("leader tx set build costs", "[herder][.][benchmark]") { CALLGRIND_START_INSTRUMENTATION; } - auto txSet = - EarlyNominationTestAccess::build(herder, seq, closeTime); + auto txSet = LeaderBuildTestAccess::build(herder, seq, closeTime); if (cold) { CALLGRIND_STOP_INSTRUMENTATION; @@ -8724,7 +7316,7 @@ TEST_CASE("leader tx set build costs", "[herder][.][benchmark]") // A follower validating the received set (cold: calibration // against perfnet's herder.txset.validate). auto const& lastSet = - EarlyNominationTestAccess::build(herder, seq, closeTime); + LeaderBuildTestAccess::build(herder, seq, closeTime); for (uint32_t rep = 0; rep < reps; ++rep) { auto received = TxSetXDRFrame::makeFromWire( @@ -8764,23 +7356,22 @@ TEST_CASE("leader tx set build costs", "[herder][.][benchmark]") auto& buildMisses = metrics.NewMeter( {"herder", "txset", "build-sig-cache-miss"}, "lookup"); auto const lastSet = - EarlyNominationTestAccess::build(herder, seq, closeTime); + LeaderBuildTestAccess::build(herder, seq, closeTime); uint64_t misses = 0; uint64_t validateMisses = 0; for (uint32_t rep = 0; rep < reps; ++rep) { PubKeyUtils::clearVerifySigCache(); auto const durationBefore = prewarmDuration.sum(); - EarlyNominationTestAccess::prewarm(herder, seq); - REQUIRE(EarlyNominationTestAccess::waitForPrewarm(herder)); + LeaderBuildTestAccess::prewarm(herder, seq); + REQUIRE(LeaderBuildTestAccess::waitForPrewarm(herder)); for (auto const& phase : phases) { metrics.NewTimer({"herder", "txset", phase}).Clear(); } auto const missesBefore = buildMisses.count(); // The build stops the (finished) prewarm and records it. - auto txSet = - EarlyNominationTestAccess::build(herder, seq, closeTime); + auto txSet = LeaderBuildTestAccess::build(herder, seq, closeTime); REQUIRE(txSet->sizeTxTotal() == std::min(sorobanCap, pull)); misses = buildMisses.count() - missesBefore; samples["prewarm (background, before the build)"].push_back( @@ -8794,8 +7385,8 @@ TEST_CASE("leader tx set build costs", "[herder][.][benchmark]") auto received = TxSetXDRFrame::makeFromWire( GeneralizedTransactionSet(lastSet->getGeneralizedXDR())); PubKeyUtils::clearVerifySigCache(); - EarlyNominationTestAccess::prewarm(herder, seq); - REQUIRE(EarlyNominationTestAccess::waitForPrewarm(herder)); + LeaderBuildTestAccess::prewarm(herder, seq); + REQUIRE(LeaderBuildTestAccess::waitForPrewarm(herder)); uint64_t hits0 = 0, misses0 = 0, hits1 = 0, misses1 = 0; PubKeyUtils::getVerifySigCacheCounts(hits0, misses0); auto const start = std::chrono::steady_clock::now(); diff --git a/src/herder/test/LeaderRoundsTests.cpp b/src/herder/test/LeaderRoundsTests.cpp new file mode 100644 index 0000000000..ad1db120e1 --- /dev/null +++ b/src/herder/test/LeaderRoundsTests.cpp @@ -0,0 +1,221 @@ +// Copyright 2026 Stellar Development Foundation and contributors. Licensed +// under the Apache License, Version 2.0. See the COPYING file at the root +// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 + +// Network-level tests of consensus without nomination: the leader of each +// round comes from a deterministic schedule seeded by the hash of the last +// closed ledger header (see herder/LeaderSchedule.h). + +#include "crypto/SHA.h" +#include "herder/Herder.h" +#include "herder/HerderImpl.h" +#include "herder/LeaderSchedule.h" +#include "ledger/LedgerHeaderUtils.h" +#include "ledger/LedgerManager.h" +#include "main/Application.h" +#include "simulation/Simulation.h" +#include "simulation/Topologies.h" +#include "test/Catch2.h" +#include "test/test.h" +#include "util/Logging.h" +#include "util/MetricsRegistry.h" +#include "util/XDROperators.h" +#include +#include +#include + +using namespace stellar; + +namespace +{ +Config +acceleratedConfig(int configNum) +{ + auto cfg = getTestConfig(configNum); + cfg.ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING = true; + return cfg; +} + +struct ClosedLedger +{ + Hash mPreviousHash; + NodeID mProposer; +}; + +// Cranks until `app` closed `lastLedger`, recording for every ledger it +// closes on the way the hash of the ledger before it and the node that signed +// the externalized value (its proposer). +std::map +crankAndRecord(Simulation::pointer const& simulation, + Application::pointer const& app, uint32_t lastLedger, + std::chrono::seconds timeout) +{ + std::map res; + auto& lm = app->getLedgerManager(); + auto prev = lm.getLastClosedLedgerHeader(); + auto const deadline = std::chrono::steady_clock::now() + timeout; + while (lm.getLastClosedLedgerNum() < lastLedger) + { + REQUIRE(std::chrono::steady_clock::now() < deadline); + simulation->crankForAtMost(std::chrono::milliseconds(20), false); + auto cur = lm.getLastClosedLedgerHeader(); + if (cur.header.ledgerSeq == prev.header.ledgerSeq + 1) + { + REQUIRE(cur.header.previousLedgerHash == prev.hash); + res[cur.header.ledgerSeq] = ClosedLedger{ + prev.hash, getLcValueSignature(cur.header.scpValue).nodeID}; + } + prev = cur; + } + return res; +} + +int64_t +leaderTimeouts(Application::pointer const& app) +{ + return static_cast( + app->getMetrics().NewHistogram({"scp", "timeout", "leader"}).sum()); +} + +int64_t +ballotTimeouts(Application::pointer const& app) +{ + return static_cast( + app->getMetrics().NewHistogram({"scp", "timeout", "prepare"}).sum()); +} +} + +TEST_CASE("leader schedule drives consensus on a healthy network", + "[herder][leader-schedule][simulation]") +{ + Hash networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); + auto simulation = Topologies::core(4, 0.75, networkID, acceleratedConfig); + simulation->startAllNodes(); + auto nodes = simulation->getNodes(); + auto nodeIDs = simulation->getNodeIDs(); + + uint32_t const lastLedger = 16; + auto closed = crankAndRecord(simulation, nodes[0], lastLedger, + std::chrono::seconds(300)); + simulation->crankUntil( + [&]() { return simulation->haveAllExternalized(lastLedger, 3); }, + std::chrono::seconds(60), false); + + int64_t timeouts = 0; + for (auto const& n : nodes) + { + timeouts += leaderTimeouts(n) + ballotTimeouts(n); + } + + // Every node computes the same schedule from the previous ledger's hash, + // and the value that externalizes is the one its round 1 leader proposed + // (unless a round timed out somewhere, e.g. on a busy test machine). + std::set proposers; + int64_t notFirstLeader = 0; + REQUIRE(closed.size() >= 10); + for (auto const& c : closed) + { + auto schedule = computeLeaderSchedule(c.second.mPreviousHash, nodeIDs); + REQUIRE(schedule.size() == 4); + REQUIRE(std::find(schedule.begin(), schedule.end(), + c.second.mProposer) != schedule.end()); + if (c.second.mProposer != schedule[0]) + { + ++notFirstLeader; + } + proposers.insert(c.second.mProposer); + } + INFO(fmt::format("ledgers {} not-first-leader {} timeouts {}", + closed.size(), notFirstLeader, timeouts)); + REQUIRE(notFirstLeader <= timeouts); + // most ledgers close in round 1 + REQUIRE(notFirstLeader * 2 < static_cast(closed.size())); + // leadership rotates + REQUIRE(proposers.size() >= 2); + + // the nodes agree on the schedule of the next ledger + auto const next = nodes[0]->getLedgerManager().getLastClosedLedgerNum() + 1; + std::optional> expected; + for (auto const& n : nodes) + { + if (n->getLedgerManager().getLastClosedLedgerNum() + 1 != next) + { + continue; + } + auto& driver = + static_cast(n->getHerder()).getHerderSCPDriver(); + auto schedule = driver.getLeaderSchedule(next); + REQUIRE(schedule.size() == 4); + if (!expected) + { + expected = schedule; + } + REQUIRE(*expected == schedule); + REQUIRE(driver.getLeaderSchedule(next + 1).empty()); + REQUIRE(driver.getLeaderSchedule(next - 1).empty()); + } +} + +TEST_CASE("leader schedule fails over when a validator is down", + "[herder][leader-schedule][simulation]") +{ + // 3 validators, threshold 2, one never started. Whenever the dead node + // leads round 1 the others time out and the round 2 leader takes over. + Hash networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); + auto simulation = + std::make_shared(networkID, acceleratedConfig); + + std::vector keys; + for (int i = 0; i < 3; i++) + { + keys.push_back( + SecretKey::fromSeed(sha256("NODE_SEED_" + std::to_string(i)))); + } + SCPQuorumSet qSet; + qSet.threshold = 2; + std::vector allNodes; + for (auto const& k : keys) + { + qSet.validators.push_back(k.getPublicKey()); + allNodes.push_back(k.getPublicKey()); + } + NodeID const dead = keys[2].getPublicKey(); + + simulation->addNode(keys[0], qSet); + simulation->addNode(keys[1], qSet); + simulation->addPendingConnection(keys[0].getPublicKey(), + keys[1].getPublicKey()); + simulation->startAllNodes(); + auto nodes = simulation->getNodes(); + + uint32_t const lastLedger = 25; + auto closed = crankAndRecord(simulation, nodes[0], lastLedger, + std::chrono::seconds(600)); + simulation->crankUntil( + [&]() { return simulation->haveAllExternalized(lastLedger, 3); }, + std::chrono::seconds(60), false); + + int64_t deadLedRound1 = 0; + REQUIRE(closed.size() >= 20); + for (auto const& c : closed) + { + // the schedule covers the whole quorum set, including the dead node + auto schedule = computeLeaderSchedule(c.second.mPreviousHash, allNodes); + REQUIRE(schedule.size() == 3); + // every ledger closed anyway, with a live node's proposal + REQUIRE(c.second.mProposer != dead); + if (schedule[0] == dead) + { + ++deadLedRound1; + } + } + INFO(fmt::format("ledgers {} led by the dead node {}", closed.size(), + deadLedRound1)); + // 1/3 of the ledgers on average: (2/3)^20 that it never happens + REQUIRE(deadLedRound1 > 0); + // and each of those cost the live nodes a leader timeout + for (auto const& n : nodes) + { + REQUIRE(leaderTimeouts(n) >= deadLedRound1); + } +} diff --git a/src/herder/test/LeaderScheduleTests.cpp b/src/herder/test/LeaderScheduleTests.cpp new file mode 100644 index 0000000000..c5bc3d1496 --- /dev/null +++ b/src/herder/test/LeaderScheduleTests.cpp @@ -0,0 +1,123 @@ +// Copyright 2026 Stellar Development Foundation and contributors. Licensed +// under the Apache License, Version 2.0. See the COPYING file at the root +// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 + +#include "crypto/SHA.h" +#include "crypto/SecretKey.h" +#include "herder/LeaderSchedule.h" +#include "test/Catch2.h" +#include "util/Math.h" +#include "util/XDROperators.h" +#include +#include +#include +#include + +using namespace stellar; + +namespace +{ +std::vector +makeNodes(size_t n) +{ + std::vector nodes; + for (size_t i = 0; i < n; i++) + { + nodes.emplace_back( + SecretKey::fromSeed(sha256(fmt::format("SCHEDULE_NODE_{}", i))) + .getPublicKey()); + } + return nodes; +} +} + +TEST_CASE("leader schedule is a deterministic permutation", + "[herder][leader-schedule]") +{ + auto nodes = makeNodes(30); + Hash seed = sha256("ledger header 41"); + + auto schedule = computeLeaderSchedule(seed, nodes); + + SECTION("it is a permutation of the node set") + { + REQUIRE(schedule.size() == nodes.size()); + REQUIRE(std::set(schedule.begin(), schedule.end()) == + std::set(nodes.begin(), nodes.end())); + } + SECTION("it does not depend on the order nodes are listed in") + { + // every node enumerates its quorum in its own order + for (int i = 0; i < 20; i++) + { + auto shuffled = nodes; + stellar::shuffle(shuffled.begin(), shuffled.end(), + getGlobalRandomEngine()); + REQUIRE(computeLeaderSchedule(seed, shuffled) == schedule); + } + } + SECTION("duplicates are ignored") + { + auto dup = nodes; + dup.insert(dup.end(), nodes.begin(), nodes.begin() + 7); + REQUIRE(computeLeaderSchedule(seed, dup) == schedule); + } + SECTION("the seed changes the order") + { + auto other = computeLeaderSchedule(sha256("ledger header 42"), nodes); + REQUIRE(std::set(other.begin(), other.end()) == + std::set(nodes.begin(), nodes.end())); + REQUIRE(other != schedule); + } + SECTION("the node set changes the order only by the nodes involved") + { + // removing a node leaves the relative order of the others alone + auto fewer = nodes; + NodeID removed = fewer.back(); + fewer.pop_back(); + auto expected = schedule; + expected.erase(std::find(expected.begin(), expected.end(), removed)); + REQUIRE(computeLeaderSchedule(seed, fewer) == expected); + } + SECTION("degenerate sets") + { + REQUIRE(computeLeaderSchedule(seed, {}).empty()); + REQUIRE(computeLeaderSchedule(seed, {nodes[3]}) == + std::vector{nodes[3]}); + } +} + +TEST_CASE("leader schedule is fair across seeds", "[herder][leader-schedule]") +{ + // Each node should lead round 1 about 1/n of the time, and the node + // leading round 2 should be independent of the one leading round 1. + size_t const n = 10; + size_t const slots = 20000; + auto nodes = makeNodes(n); + std::map first; + std::map, size_t> firstTwo; + for (size_t i = 0; i < slots; i++) + { + auto schedule = + computeLeaderSchedule(sha256(fmt::format("header {}", i)), nodes); + first[schedule[0]]++; + firstTwo[{schedule[0], schedule[1]}]++; + } + REQUIRE(first.size() == n); + double const expected = static_cast(slots) / n; + for (auto const& f : first) + { + // 2000 expected, standard deviation ~42: 10% is over 4 sigma + REQUIRE(f.second > expected * 0.9); + REQUIRE(f.second < expected * 1.1); + } + // all ordered pairs of distinct nodes show up + REQUIRE(firstTwo.size() == n * (n - 1)); + double const expectedPair = static_cast(slots) / (n * (n - 1)); + for (auto const& f : firstTwo) + { + // 222 expected, standard deviation ~15 + REQUIRE(f.second > expectedPair * 0.6); + REQUIRE(f.second < expectedPair * 1.4); + } +} diff --git a/src/herder/test/PendingEnvelopesTests.cpp b/src/herder/test/PendingEnvelopesTests.cpp index 0eee4f71c6..6b8085ade0 100644 --- a/src/herder/test/PendingEnvelopesTests.cpp +++ b/src/herder/test/PendingEnvelopesTests.cpp @@ -100,6 +100,44 @@ makePublicKey(int i) auto secretKey = SecretKey::fromSeed(hash); return secretKey.getPublicKey(); } + +// The local node votes for the round 1 leader's ballot as soon as SCP sees +// it, before its own trigger. These tests feed the node a peer's statements +// and then inspect PendingEnvelopes (reference counts, fetch status), which +// such a vote would change. So they use a peer that does not lead round 1 of +// the next slot: the local node then leads it (or a node that never speaks +// does, once the peer's quorum set is known) and does not vote. +struct FollowerPeerApp +{ + // destroyed last: the application uses the clock + std::unique_ptr mClock; + SecretKey mPeer; + Application::pointer mApp; +}; + +FollowerPeerApp +createAppWithFollowerPeer(Config const& baseCfg) +{ + for (int i = 0; i < 100; ++i) + { + FollowerPeerApp res; + res.mPeer = SecretKey::pseudoRandomForTesting(); + Config cfg(baseCfg); + cfg.QUORUM_SET.validators.emplace_back(res.mPeer.getPublicKey()); + res.mClock = std::make_unique(); + res.mApp = createTestApplication(*res.mClock, cfg); + auto& herder = static_cast(res.mApp->getHerder()); + auto const seq = + res.mApp->getLedgerManager().getLastClosedLedgerNum() + 1; + auto leader = herder.getHerderSCPDriver().getRoundLeader(seq, 1); + if (leader && !(*leader == res.mPeer.getPublicKey())) + { + return res; + } + } + throw std::runtime_error( + "could not pick a peer that does not lead round 1"); +} } // namespace TEST_CASE_VERSIONS("PendingEnvelopes recvSCPEnvelope", "[herder]") @@ -108,12 +146,10 @@ TEST_CASE_VERSIONS("PendingEnvelopes recvSCPEnvelope", "[herder]") cfg.MANUAL_CLOSE = false; cfg.EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD = true; - VirtualClock clock; - - auto s = SecretKey::pseudoRandomForTesting(); + auto fixture = createAppWithFollowerPeer(cfg); + auto s = fixture.mPeer; auto& pk = s.getPublicKey(); - cfg.QUORUM_SET.validators.emplace_back(s.getPublicKey()); - Application::pointer app = createTestApplication(clock, cfg); + Application::pointer app = fixture.mApp; auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); @@ -766,12 +802,10 @@ TEST_CASE("PendingEnvelopes recvSCPEnvelope without parallel tx set download", cfg.MANUAL_CLOSE = false; cfg.EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD = false; - VirtualClock clock; - - auto s = SecretKey::pseudoRandomForTesting(); + auto fixture = createAppWithFollowerPeer(cfg); + auto s = fixture.mPeer; auto& pk = s.getPublicKey(); - cfg.QUORUM_SET.validators.emplace_back(s.getPublicKey()); - Application::pointer app = createTestApplication(clock, cfg); + Application::pointer app = fixture.mApp; auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); @@ -822,11 +856,11 @@ TEST_CASE("PendingEnvelopes re-requests tx sets the overlay does not deliver", "[herder]") { Config cfg(getTestConfig()); - VirtualClock clock; - auto s = SecretKey::pseudoRandomForTesting(); + auto fixture = createAppWithFollowerPeer(cfg); + auto& clock = *fixture.mClock; + auto s = fixture.mPeer; auto& pk = s.getPublicKey(); - cfg.QUORUM_SET.validators.emplace_back(pk); - Application::pointer app = createTestApplication(clock, cfg); + Application::pointer app = fixture.mApp; auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); auto& herder = static_cast(app->getHerder()); auto& pendingEnvelopes = herder.getPendingEnvelopes(); diff --git a/src/herder/test/QuorumTrackerTests.cpp b/src/herder/test/QuorumTrackerTests.cpp index 030f800824..04abe00560 100644 --- a/src/herder/test/QuorumTrackerTests.cpp +++ b/src/herder/test/QuorumTrackerTests.cpp @@ -78,21 +78,25 @@ testQuorumTracker() herder->recvTxSet(p.mTxSet->getContentsHash(), p.mTxSet); } }; - auto recvNom = [&](uint64 slotID, SecretKey const& k, - SCPQuorumSet const& qSet, - std::vector const& pp) { + auto recvPrepare = [&](uint64 slotID, SecretKey const& k, + SCPQuorumSet const& qSet, + std::vector const& pp) { + // A PREPARE that references every value of `pp`: the last one is the + // ballot's value, an earlier one (if any) is the prepared value. More + // values make a newer statement. + releaseAssert(!pp.empty() && pp.size() <= 2); SCPEnvelope envelope; - envelope.statement.pledges.type(SCP_ST_NOMINATE); - auto& nom = envelope.statement.pledges.nominate(); - - std::set values; - for (auto& p : pp) + envelope.statement.pledges.type(SCP_ST_PREPARE); + auto& prep = envelope.statement.pledges.prepare(); + prep.ballot.counter = static_cast(pp.size()); + prep.ballot.value = pp.back().mSignedV; + if (pp.size() > 1) { - values.insert(p.mSignedV); + prep.prepared.activate().counter = 1; + prep.prepared->value = pp.front().mSignedV; } - nom.votes.insert(nom.votes.begin(), values.begin(), values.end()); auto qSetH = sha256(xdr::xdr_to_opaque(qSet)); - nom.quorumSetHash = qSetH; + prep.quorumSetHash = qSetH; recvEnvelope(envelope, slotID, k, qSet, pp); }; auto recvExternalize = [&](uint64 slotID, SecretKey const& k, @@ -135,17 +139,17 @@ testQuorumTracker() SECTION("Receive self") { checkInQuorum({0, 1}); - recvNom(3, cfg.NODE_SEED, cfg.QUORUM_SET, {vv}); + recvPrepare(3, cfg.NODE_SEED, cfg.QUORUM_SET, {vv}); checkInQuorum({0, 1}); } SECTION("Expand 0") { checkInQuorum({0, 1}); - recvNom(3, otherKeys[0], qSet0, {vv}); + recvPrepare(3, otherKeys[0], qSet0, {vv}); checkInQuorum({0, 1, 2, 3}); SECTION("Expand 2") { - recvNom(3, otherKeys[2], qSet2, {vv}); + recvPrepare(3, otherKeys[2], qSet2, {vv}); checkInQuorum({0, 1, 2, 3, 5, 6}); SECTION("node restart") { @@ -168,13 +172,13 @@ testQuorumTracker() SECTION("Update 0's qSet") { auto vv2 = makeValue(2); - recvNom(3, otherKeys[0], qSet0b, {vv, vv2}); + recvPrepare(3, otherKeys[0], qSet0b, {vv, vv2}); checkInQuorum({0, 1, 4, 5}); } SECTION("Update 0's qSet in an old slot") { auto vv2 = makeValue(2); - recvNom(2, otherKeys[0], qSet0b, {vv, vv2}); + recvPrepare(2, otherKeys[0], qSet0b, {vv, vv2}); // nothing changes (slot 3 has precedence) checkInQuorum({0, 1, 2, 3}); } diff --git a/src/main/Config.cpp b/src/main/Config.cpp index 457e23ab9c..33e580961a 100644 --- a/src/main/Config.cpp +++ b/src/main/Config.cpp @@ -68,7 +68,6 @@ static std::unordered_set const TESTING_ONLY_OPTIONS = { "ARTIFICIALLY_SET_SURVEY_PHASE_DURATION_FOR_TESTING", "ARTIFICIALLY_DELAY_BUCKET_APPLICATION_FOR_TESTING", "ARTIFICIALLY_SLEEP_MAIN_THREAD_FOR_TESTING", - "ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING", "ARTIFICIALLY_SKIP_CONNECTION_ADJUSTMENT_FOR_TESTING", "ARTIFICIALLY_DELAY_LEDGER_CLOSE_FOR_TESTING", "SKIP_HIGH_CRITICAL_VALIDATOR_CHECKS_FOR_TESTING", @@ -155,7 +154,7 @@ Config::Config() : NODE_SEED(SecretKey::random()) TESTING_MAX_CLASSIC_BYTE_ALLOWANCE = 0; IGNORE_MESSAGE_LIMITS_FOR_TESTING = false; TESTING_IGNORE_LEDGER_TIME_UPGRADE_BOUNDS = false; - TESTING_NOMINATE_RANDOM_VALUES = false; + TESTING_PROPOSE_RANDOM_VALUES = false; ALLOW_PRIVATE_ADDRESSES_FOR_TESTING = false; #endif @@ -376,8 +375,6 @@ Config::Config() : NODE_SEED(SecretKey::random()) SKIP_HIGH_CRITICAL_VALIDATOR_CHECKS_FOR_TESTING = false; ARTIFICIALLY_SET_SYSTEM_CLOCK_OFFSET_FOR_TESTING = std::chrono::milliseconds::zero(); - ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING = - std::chrono::milliseconds::zero(); #endif #ifdef BEST_OFFER_DEBUGGING @@ -1179,8 +1176,8 @@ Config::processConfig(std::shared_ptr t) "upgrade bounds for testing"); } }}, - {"TESTING_NOMINATE_RANDOM_VALUES", - [&]() { TESTING_NOMINATE_RANDOM_VALUES = readBool(item); }}, + {"TESTING_PROPOSE_RANDOM_VALUES", + [&]() { TESTING_PROPOSE_RANDOM_VALUES = readBool(item); }}, #endif {"PEER_PORT", [&]() { PEER_PORT = readInt(item, 1); }}, @@ -1985,11 +1982,6 @@ Config::processConfig(std::shared_ptr t) ARTIFICIALLY_SET_SYSTEM_CLOCK_OFFSET_FOR_TESTING = std::chrono::milliseconds(readInt(item)); }}, - {"ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING", - [&]() { - ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING = - std::chrono::milliseconds(readInt(item)); - }}, #endif {"MAX_DEX_TX_OPERATIONS_IN_TX_SET", [&]() { diff --git a/src/main/Config.h b/src/main/Config.h index da3a797206..f3238b9a41 100644 --- a/src/main/Config.h +++ b/src/main/Config.h @@ -952,16 +952,12 @@ class Config : public std::enable_shared_from_this // When set, this node will nominate random values when it is the round // leader. This is useful for testing CAP-0083 behavior. This is a testing // only flag. - bool TESTING_NOMINATE_RANDOM_VALUES; + bool TESTING_PROPOSE_RANDOM_VALUES; // Injects a signed wall-clock offset into the node's system clock for // testing. Expressed in milliseconds. std::chrono::milliseconds ARTIFICIALLY_SET_SYSTEM_CLOCK_OFFSET_FOR_TESTING; - // Delay emission of updated nomination messages for testing nomination - // timeout behavior. Expressed in milliseconds. - std::chrono::milliseconds ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING; - // Set QUORUM_SET using automatic quorum set configuration based on // `validators`. void diff --git a/src/overlay/test/OverlayIPCTests.cpp b/src/overlay/test/OverlayIPCTests.cpp index 12a3f5c60c..e79cd21b08 100644 --- a/src/overlay/test/OverlayIPCTests.cpp +++ b/src/overlay/test/OverlayIPCTests.cpp @@ -56,18 +56,19 @@ makeMockSCPEnvelope(uint64_t slotIndex, uint32_t nodeId) { SCPEnvelope env; env.statement.slotIndex = slotIndex; - env.statement.pledges.type(SCP_ST_NOMINATE); + env.statement.pledges.type(SCP_ST_PREPARE); // Set some mock data - auto& nom = env.statement.pledges.nominate(); - nom.quorumSetHash.fill(static_cast(nodeId)); + auto& prep = env.statement.pledges.prepare(); + prep.quorumSetHash.fill(static_cast(nodeId)); + prep.ballot.counter = 1; // Value is opaque<> (xvector), not Hash Value mockValue; mockValue.resize(32); std::fill(mockValue.begin(), mockValue.end(), static_cast(slotIndex & 0xFF)); - nom.votes.push_back(mockValue); + prep.ballot.value = mockValue; return env; } @@ -404,11 +405,11 @@ TEST_CASE("Rust overlay SCP consensus", "[overlay-ipc][.]") } /** - * Test TX set building and nomination hash request. + * Test TX set building and proposal hash request. * * This test: * 1. Creates an OverlayIPC connection to a Rust overlay - * 2. Requests a nomination hash (which builds a TX set from empty mempool) + * 2. Requests a proposal hash (which builds a TX set from empty mempool) * 3. Verifies a hash is returned */ TEST_CASE("Rust overlay get top transactions", "[overlay-ipc][.]") @@ -1008,7 +1009,7 @@ TEST_CASE("Rust overlay TX included in ledger", "[overlay-ipc][.]") * - 200 tx/ledger (heavy load) * * For each rate, runs for several ledgers and measures: - * - scp.timing.nominated (time from nomination to prepare) + * - scp.timing.trigger-to-ballot (time from the trigger to the first ballot) * - scp.timing.externalized (time from prepare to externalize) * - ledger.ledger.close (total ledger close time) */ @@ -1034,8 +1035,8 @@ TEST_CASE("Rust overlay SCP latency under TX load", std::string label; int txSubmitted; int txIncluded; - double scpNominatedMean; - double scpNominatedMax; + double scpTriggerToBallotMean; + double scpTriggerToBallotMax; double scpExternalizedMean; double scpExternalizedMax; double ledgerCloseMean; @@ -1095,8 +1096,8 @@ TEST_CASE("Rust overlay SCP latency under TX load", // Get metrics (they accumulate across ledgers) auto& metrics = node0->getMetrics(); - auto& scpNominated = - metrics.NewTimer({"scp", "timing", "nominated"}); + auto& scpTriggerToBallot = + metrics.NewTimer({"scp", "timing", "trigger-to-ballot"}); auto& scpExternalized = metrics.NewTimer({"scp", "timing", "externalized"}); auto& ledgerClose = metrics.NewTimer({"ledger", "ledger", "close"}); @@ -1214,8 +1215,8 @@ TEST_CASE("Rust overlay SCP latency under TX load", res.label = run.label; res.txSubmitted = txSubmitted; res.txIncluded = static_cast(txIncluded); - res.scpNominatedMean = scpNominated.mean(); - res.scpNominatedMax = scpNominated.max(); + res.scpTriggerToBallotMean = scpTriggerToBallot.mean(); + res.scpTriggerToBallotMax = scpTriggerToBallot.max(); res.scpExternalizedMean = scpExternalized.mean(); res.scpExternalizedMax = scpExternalized.max(); res.ledgerCloseMean = ledgerClose.mean(); @@ -1250,8 +1251,8 @@ TEST_CASE("Rust overlay SCP latency under TX load", { LOG_INFO(DEFAULT_LOG, "{:<20} {:>10} {:>10} {:>12.2f} {:>12.2f} {:>12.2f} {:>12.2f}", - r.label, r.txSubmitted, r.txIncluded, r.scpNominatedMean, - r.scpNominatedMax, r.scpExternalizedMean, + r.label, r.txSubmitted, r.txIncluded, r.scpTriggerToBallotMean, + r.scpTriggerToBallotMax, r.scpExternalizedMean, r.scpExternalizedMax); } @@ -1272,11 +1273,11 @@ TEST_CASE("Rust overlay SCP latency under TX load", // Allow 3x degradation from light to heavy load if (allResults.size() >= 2) { - double lightMean = allResults[0].scpNominatedMean; - double heavyMean = allResults.back().scpNominatedMean; + double lightMean = allResults[0].scpTriggerToBallotMean; + double heavyMean = allResults.back().scpTriggerToBallotMean; LOG_INFO(DEFAULT_LOG, - "SCP nominated latency ratio (heavy/light): {:.2f}x", + "SCP trigger-to-ballot latency ratio (heavy/light): {:.2f}x", heavyMean / lightMean); // Warn but don't fail if degradation is significant @@ -1400,7 +1401,8 @@ TEST_CASE("Rust overlay 15-node 2000 TPS stress test", // Get metrics from node 0 auto& metrics = nodes[0]->getMetrics(); - auto& scpNominated = metrics.NewTimer({"scp", "timing", "nominated"}); + auto& scpTriggerToBallot = + metrics.NewTimer({"scp", "timing", "trigger-to-ballot"}); auto& scpExternalized = metrics.NewTimer({"scp", "timing", "externalized"}); auto& ledgerClose = metrics.NewTimer({"ledger", "ledger", "close"}); @@ -1470,8 +1472,8 @@ TEST_CASE("Rust overlay 15-node 2000 TPS stress test", LOG_INFO(DEFAULT_LOG, " Effective TPS: {:.0f}", effectiveTps); LOG_INFO(DEFAULT_LOG, ""); LOG_INFO(DEFAULT_LOG, "SCP Timing (ms):"); - LOG_INFO(DEFAULT_LOG, " Nominated: mean={:.2f}, max={:.2f}", - scpNominated.mean(), scpNominated.max()); + LOG_INFO(DEFAULT_LOG, " TriggerToBallot: mean={:.2f}, max={:.2f}", + scpTriggerToBallot.mean(), scpTriggerToBallot.max()); LOG_INFO(DEFAULT_LOG, " Externalized: mean={:.2f}, max={:.2f}", scpExternalized.mean(), scpExternalized.max()); LOG_INFO(DEFAULT_LOG, ""); @@ -1484,10 +1486,10 @@ TEST_CASE("Rust overlay 15-node 2000 TPS stress test", // Warn if SCP latency is concerning (but don't fail - this is // informational) - if (scpNominated.max() > 5000) + if (scpTriggerToBallot.max() > 5000) { - WARN("SCP nomination max latency exceeded 5s: " << scpNominated.max() - << "ms"); + WARN("SCP trigger-to-ballot max latency exceeded 5s: " + << scpTriggerToBallot.max() << "ms"); } LOG_INFO(DEFAULT_LOG, diff --git a/src/scp/BallotProtocol.cpp b/src/scp/BallotProtocol.cpp index b69f5c2a86..01bc51c566 100644 --- a/src/scp/BallotProtocol.cpp +++ b/src/scp/BallotProtocol.cpp @@ -32,6 +32,11 @@ BallotProtocol::BallotProtocol(Slot& slot) , mPhase(SCP_PHASE_PREPARE) , mCurrentMessageLevel(0) , mTimerExpCount(0) + , mLeaderRound(0) + , mLeaderRoundsStarted(false) + , mLeaderTimerRound(0) + , mLeaderTimerExtended(false) + , mInProgressRound(false) { } @@ -247,7 +252,18 @@ BallotProtocol::processEnvelope(SCPEnvelopeWrapperPtr envelope, bool self) } recordEnvelope(envelope); + if (!self) + { + // the statement may be the round leader's proposal + progressRound(); + } advanceSlot(statement); + if (!self) + { + // processing the statement may have moved the local node to a + // new round, or given it a value it has to stick to + progressRound(); + } return SCP::EnvelopeState::VALID; } @@ -349,28 +365,331 @@ BallotProtocol::abandonBallot(uint32 n) { ZoneScoped; CLOG_TRACE(SCP, "BallotProtocol::abandonBallot"); + if (mPhase != SCP_PHASE_PREPARE && mPhase != SCP_PHASE_CONFIRM) + { + return false; + } + if (n == 0 && mLeaderRound == UINT32_MAX) + { + // already at the "infinite" counter of a committed ballot + return false; + } + uint32 newRound = (n == 0) ? (mLeaderRound + 1) : n; + if (newRound > mLeaderRound) + { + mLeaderRound = newRound; + if (mLeaderRound != UINT32_MAX) + { + mSlot.getSCPDriver().enteredRound(mSlot.getSlotIndex(), + mLeaderRound, n == 0); + } + } + return progressRound(); +} + +bool +BallotProtocol::startLeaderRounds(ProposalValueSupplier const& makeValue) +{ + ZoneScoped; + if (mPhase != SCP_PHASE_PREPARE && mPhase != SCP_PHASE_CONFIRM) + { + return false; + } + mMakeValue = makeValue; + mLeaderRoundsStarted = true; + if (mLeaderRound == 0) + { + mLeaderRound = 1; + mSlot.getSCPDriver().enteredRound(mSlot.getSlotIndex(), mLeaderRound, + false); + } + return progressRound(); +} + +std::optional +BallotProtocol::getLeaderBallot(NodeID const& leader, uint32 round) const +{ + auto it = mLatestEnvelopes.find(leader); + if (it == mLatestEnvelopes.end()) + { + return std::nullopt; + } + auto const& pl = it->second->getStatement().pledges; + switch (pl.type()) + { + case SCP_ST_PREPARE: + if (pl.prepare().ballot.counter == round) + { + return pl.prepare().ballot; + } + break; + case SCP_ST_CONFIRM: + if (pl.confirm().ballot.counter == round) + { + return pl.confirm().ballot; + } + break; + default: + // an EXTERNALIZE statement is not tied to a round: the regular + // accept/confirm steps deal with it + break; + } + return std::nullopt; +} + +bool +BallotProtocol::progressRound() +{ + ZoneScoped; + if (mInProgressRound) + { + // the outer call loops until there is nothing left to do + return false; + } + mInProgressRound = true; bool res = false; - auto v = mSlot.getLatestCompositeCandidate(); + try + { + // A step that changes the ballot may enable another one (e.g. the + // node moved to a new round while processing its own statement). + // The ballot only ever increases, and each productive step raises + // its counter to the current round, so this ends. + while (progressRoundStep()) + { + res = true; + } + updateLeaderTimer(); + } + catch (...) + { + mInProgressRound = false; + throw; + } + mInProgressRound = false; + return res; +} - if (!v || v->getValue().empty()) +bool +BallotProtocol::progressRoundStep() +{ + if (mPhase != SCP_PHASE_PREPARE && mPhase != SCP_PHASE_CONFIRM) { - if (mCurrentBallot) + return false; + } + if (!getLocalNode()->isValidator()) + { + // only validators vote + return false; + } + + // Before the application starts the slot, the local node may still adopt + // what the round 1 leader put forward: followers do not wait for their + // own clock. + uint32 const round = std::max(mLeaderRound, 1); + uint32 const counter = + mCurrentBallot ? mCurrentBallot->getBallot().counter : 0; + if (counter >= round) + { + return false; + } + + auto& driver = mSlot.getSCPDriver(); + uint64 const slotIndex = mSlot.getSlotIndex(); + + if (mValueOverride) + { + // The node saw this value confirmed prepared, or voted to commit it: + // it sticks to it whoever leads the round (bumpState enforces this + // for any value passed in). + if (mLeaderRound == 0) { - v = mCurrentBallot->getWValue(); + return false; } + return bumpState(mValueOverride->getValue(), round); } - if (v && !v->getValue().empty()) + + if (mPhase != SCP_PHASE_PREPARE) { - if (n == 0) + // CONFIRM has a value override + return false; + } + + if (round == UINT32_MAX) + { + // The "infinite" counter of a committed ballot (a v-blocking set + // externalized) is not a round and has no leader: just keep up with + // the value the node already votes for, if any. + if (mCurrentBallot && mLeaderRound != 0) { - res = bumpState(v->getValue(), true); + Value v = mCurrentBallot->getBallot().value; + return bumpState(v, round); } - else + return false; + } + + auto leader = driver.getRoundLeader(slotIndex, round); + if (!leader) + { + return false; + } + + if (*leader == getLocalNode()->getNodeID()) + { + if (mLeaderRound == 0) + { + // not started: the application is not ready to propose + return false; + } + // Keep the value the node already votes for unless it is not + // acceptable as a proposal (e.g. an empty-tx-set value): the nodes + // that voted for it already know (or are fetching) its transaction + // set. bumpState gives up on a transaction set that never arrived. + if (mCurrentBallot && + driver.validateValue(slotIndex, mCurrentBallot->getBallot().value, + /*proposal=*/true) != SCPDriver::kInvalidValue) { - res = bumpState(v->getValue(), n); + Value v = mCurrentBallot->getBallot().value; + driver.proposedValue(slotIndex, round, v); + return bumpState(v, round); } + if (!mLeaderRoundsStarted || !mMakeValue || mCurrentMessageLevel != 0) + { + // Building a value re-enters the slot, so it is only done from + // the top level; processEnvelope calls back once it is there. + return false; + } + auto value = mMakeValue(); + // Building the value can make pending envelopes ready and re-enter + // SCP: the slot may have moved on. + uint32 const newCounter = + mCurrentBallot ? mCurrentBallot->getBallot().counter : 0; + if (!value || mPhase != SCP_PHASE_PREPARE || mValueOverride || + mLeaderRound != round || newCounter >= round) + { + // re-evaluate from the new state, if it changed + return newCounter != counter || mLeaderRound != round || + mValueOverride != nullptr; + } + driver.proposedValue(slotIndex, round, value->getValue()); + return bumpState(value->getValue(), round); } - return res; + + uint32 adoptRound = round; + auto leaderBallot = getLeaderBallot(*leader, round); + if (!leaderBallot && !mCurrentBallot) + { + // The node gave up on earlier rounds without ever voting. If one of + // their leaders' proposals shows up late (e.g. its transaction set + // took longer than the leader timeout to arrive), vote for it at its + // own counter rather than discard it: the nodes that were in time + // voted for it too. The node stays in its round, so the current + // leader's ballot (higher counter) still takes over when it comes. + static uint32 const MAX_LATE_ROUNDS = 8; + for (uint32 r = round - 1; + r >= 1 && round - r <= MAX_LATE_ROUNDS && !leaderBallot; --r) + { + auto pastLeader = driver.getRoundLeader(slotIndex, r); + if (pastLeader && !(*pastLeader == getLocalNode()->getNodeID())) + { + leaderBallot = getLeaderBallot(*pastLeader, r); + adoptRound = r; + } + } + } + if (!leaderBallot) + { + return false; + } + // Voting for a leader's value is the local node's endorsement of it as a + // proposal, so the stricter validation applies (as it did for nomination + // votes): e.g. it must only carry upgrades the local node armed. + // An empty-tx-set value is never built as a proposal: a leader only votes + // for one after the ballot protocol replaced a value whose transaction + // set did not arrive. Following it is a ballot-level decision, so the + // ballot validation applies. + bool const proposal = !driver.isEmptyTxSetValue(leaderBallot->value); + if (driver.validateValue(slotIndex, leaderBallot->value, proposal) < + SCPDriver::kStructurallyValidValue) + { + return false; + } + return bumpState(leaderBallot->value, adoptRound); +} + +void +BallotProtocol::updateLeaderTimer() +{ + uint32 const counter = + mCurrentBallot ? mCurrentBallot->getBallot().counter : 0; + bool const waiting = mPhase == SCP_PHASE_PREPARE && mLeaderRound != 0 && + mLeaderRound != UINT32_MAX && counter < mLeaderRound && + getLocalNode()->isValidator(); + if (waiting) + { + if (mLeaderTimerRound != mLeaderRound) + { + // The ballot (if any) is for a round the node gave up on. + stopBallotProtocolTimer(); + mLeaderTimerRound = mLeaderRound; + mLeaderTimerExtended = false; + armLeaderTimer(mLeaderRound); + } + } + else if (mLeaderTimerRound != 0) + { + mLeaderTimerRound = 0; + mSlot.getSCPDriver().setupTimer(mSlot.getSlotIndex(), + Slot::LEADER_TIMER, + std::chrono::seconds::zero(), nullptr); + } +} + +void +BallotProtocol::armLeaderTimer(uint32 round) +{ + std::shared_ptr slot = mSlot.shared_from_this(); + mSlot.getSCPDriver().setupTimer( + mSlot.getSlotIndex(), Slot::LEADER_TIMER, + mSlot.getSCPDriver().computeTimeout(round), [slot, round]() { + slot->getBallotProtocol().leaderTimerExpired(round); + }); +} + +void +BallotProtocol::leaderTimerExpired(uint32 round) +{ + if (mLeaderTimerRound != round) + { + // stale callback + return; + } + uint32 const counter = + mCurrentBallot ? mCurrentBallot->getBallot().counter : 0; + if (mPhase != SCP_PHASE_PREPARE || mLeaderRound != round || + counter >= round) + { + mLeaderTimerRound = 0; + return; + } + if (!mLeaderTimerExtended) + { + // The leader is not silent if its statement already reached the + // application and is only waiting for its transaction set: give it + // one more timeout before moving on, rather than have the next leader + // build a competing value. + auto leader = + mSlot.getSCPDriver().getRoundLeader(mSlot.getSlotIndex(), round); + if (leader && mSlot.getSCPDriver().isProposalPending( + mSlot.getSlotIndex(), *leader)) + { + mLeaderTimerExtended = true; + armLeaderTimer(round); + return; + } + } + mLeaderTimerRound = 0; + mTimerExpCount++; + abandonBallot(0); } bool @@ -580,13 +899,27 @@ BallotProtocol::bumpToBallot(SCPBallot const& ballot, bool check) { mHeardFromQuorum = false; } + + // The ballot can also move through the regular accept/confirm steps + // (e.g. a quorum confirmed a higher ballot prepared). + if (ballot.counter > mLeaderRound) + { + mLeaderRound = ballot.counter; + // UINT32_MAX is the "infinite" counter of a committed ballot, not a + // round + if (mLeaderRound != UINT32_MAX) + { + mSlot.getSCPDriver().enteredRound(mSlot.getSlotIndex(), + mLeaderRound, false); + } + } } void BallotProtocol::startBallotProtocolTimer() { std::chrono::milliseconds timeout = mSlot.getSCPDriver().computeTimeout( - mCurrentBallot->getBallot().counter, /*isNomination=*/false); + mCurrentBallot->getBallot().counter); std::shared_ptr slot = mSlot.shared_from_this(); mSlot.getSCPDriver().setupTimer( @@ -607,6 +940,8 @@ void BallotProtocol::ballotProtocolTimerExpired() { mTimerExpCount++; + // The timer only runs for a ballot at the current round (see + // checkHeardFromQuorum), so this moves to the round after it. abandonBallot(0); } @@ -1438,7 +1773,7 @@ void BallotProtocol::throwIfValueInvalidForConfirmCommit(Value const& value) { auto validationLevel = mSlot.getSCPDriver().validateValue( - mSlot.getSlotIndex(), value, /*nomination=*/false); + mSlot.getSlotIndex(), value, /*proposal=*/false); if (validationLevel == SCPDriver::kFullyValidatedValue || validationLevel == SCPDriver::kMaybeValidNotCurrentValue) @@ -1685,7 +2020,7 @@ BallotProtocol::setConfirmCommit(SCPBallot const& c, SCPBallot const& h) emitCurrentStateStatement(); - mSlot.stopNomination(); + updateLeaderTimer(); mSlot.getSCPDriver().valueExternalized(mSlot.getSlotIndex(), mCommit->getBallot().value); @@ -1931,6 +2266,11 @@ BallotProtocol::setStateFromEnvelope(SCPEnvelopeWrapperPtr e) { mCommit = makeBallot(prep.nC, b.value); } + if (prep.nH) + { + // as when the node confirmed the ballot prepared + mValueOverride = mSlot.getSCPDriver().wrapValue(b.value); + } mPhase = SCP_PHASE_PREPARE; } break; @@ -1942,6 +2282,7 @@ BallotProtocol::setStateFromEnvelope(SCPEnvelopeWrapperPtr e) mPrepared = makeBallot(c.nPrepared, v); mHighBallot = makeBallot(c.nH, v); mCommit = makeBallot(c.nCommit, v); + mValueOverride = mSlot.getSCPDriver().wrapValue(v); mPhase = SCP_PHASE_CONFIRM; } break; @@ -2174,6 +2515,12 @@ BallotProtocol::getJsonInfo() ret["heard"] = mHeardFromQuorum; ret["ballot"] = ballotToStr(mCurrentBallot); ret["phase"] = phaseNames[mPhase]; + ret["round"] = mLeaderRound; + if (auto leader = mSlot.getSCPDriver().getRoundLeader( + mSlot.getSlotIndex(), std::max(mLeaderRound, 1))) + { + ret["leader"] = mSlot.getSCPDriver().toShortString(*leader); + } ret["state"] = getLocalState(); return ret; @@ -2387,7 +2734,10 @@ BallotProtocol::checkHeardFromQuorum() // if we transition from not heard -> heard, we start the timer mSlot.getSCPDriver().ballotDidHearFromQuorum( mSlot.getSlotIndex(), mCurrentBallot->getBallot()); - if (mPhase != SCP_PHASE_EXTERNALIZE) + // A ballot behind the current round is one the node already + // gave up on: the leader timer runs instead. + if (mPhase != SCP_PHASE_EXTERNALIZE && + mCurrentBallot->getBallot().counter >= mLeaderRound) { startBallotProtocolTimer(); } diff --git a/src/scp/BallotProtocol.h b/src/scp/BallotProtocol.h index 2ff2cb1207..e7126d1449 100644 --- a/src/scp/BallotProtocol.h +++ b/src/scp/BallotProtocol.h @@ -9,6 +9,7 @@ #include "util/GlobalChecks.h" #include #include +#include #include #include #include @@ -95,6 +96,31 @@ class BallotProtocol SCPEnvelopeWrapperPtr mLastEnvelopeEmit; // last envelope emitted by this node + // ** Leader rounds + // There is no nomination phase. A round is a ballot counter, and each + // round has one leader (SCPDriver::getRoundLeader). At counter r a node + // votes for the ballot `leader(r)` votes for at counter r, unless it + // already holds a value it must stick to (mValueOverride). + + // The round the local node is in: the highest counter it is willing to + // vote at. 0 until the node starts the slot or adopts a round 1 ballot. + // Invariant: mLeaderRound >= mCurrentBallot's counter (so it is + // UINT32_MAX once the node moved to the "infinite" counter of a committed + // ballot). + uint32 mLeaderRound; + // True once the application started the slot (startLeaderRounds). + bool mLeaderRoundsStarted; + // Builds the local node's own value when it leads a round. + ProposalValueSupplier mMakeValue; + // Round the leader timer is armed for (0 = not armed). + uint32 mLeaderTimerRound; + // True once the leader timer of mLeaderTimerRound got its one extension + // (the leader's proposal was pending in the application when it fired). + bool mLeaderTimerExtended; + // Guards against re-entering progressRound (the value supplier and + // bumpState both re-enter the slot). + bool mInProgressRound; + public: BallotProtocol(Slot& slot); @@ -106,10 +132,21 @@ class BallotProtocol bool self); void ballotProtocolTimerExpired(); - // abandon's current ballot, move to a new ballot - // at counter `n` (or, if n == 0, increment current counter) + void leaderTimerExpired(uint32 round); + // abandon's current ballot, move to round + // `n` (or, if n == 0, to the next round) bool abandonBallot(uint32 n); + // Called when the local node is ready to take part in consensus on this + // slot: enters round 1 (if not already further along). + bool startLeaderRounds(ProposalValueSupplier const& makeValue); + + uint32 + getLeaderRound() const + { + return mLeaderRound; + } + // bumps the ballot based on the local state and the value passed in: // in prepare phase, attempts to take value // otherwise, no-ops @@ -328,6 +365,21 @@ class BallotProtocol void stopBallotProtocolTimer(); void checkHeardFromQuorum(); + // Tries to bring the current ballot up to the current round: with the + // value the node must stick to, else as the round's leader (own value), + // else by adopting what the round's leader voted for. Arms the leader + // timer while the ballot is still behind the round. + // Returns true if the current ballot changed. + bool progressRound(); + // One attempt of the above; returns true if the current ballot changed. + bool progressRoundStep(); + // Returns the ballot `leader` voted for at exactly counter `round`, if + // its latest statement is at that counter. + std::optional getLeaderBallot(NodeID const& leader, + uint32 round) const; + void updateLeaderTimer(); + void armLeaderTimer(uint32 round); + SCPBallotWrapperUPtr makeBallot(SCPBallot const& b) const; SCPBallotWrapperUPtr makeBallot(uint32 c, Value const& v) const; diff --git a/src/scp/NominationProtocol.cpp b/src/scp/NominationProtocol.cpp deleted file mode 100644 index 015bd8f814..0000000000 --- a/src/scp/NominationProtocol.cpp +++ /dev/null @@ -1,926 +0,0 @@ -// Copyright 2014 Stellar Development Foundation and contributors. Licensed -// under the Apache License, Version 2.0. See the COPYING file at the root -// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 - -#include "NominationProtocol.h" - -#include "Slot.h" -#include "crypto/Hex.h" -#include "lib/json/json.h" -#include "scp/LocalNode.h" -#include "scp/QuorumSetUtils.h" -#include "util/GlobalChecks.h" -#include "util/Logging.h" -#include "util/XDROperators.h" -#include "xdrpp/marshal.h" -#include -#include -#include - -namespace stellar -{ -using namespace std::placeholders; - -NominationProtocol::NominationProtocol(Slot& slot) - : mSlot(slot), mRoundNumber(0), mNominationStarted(false), mTimerExpCount(0) -{ -} - -std::set -NominationProtocol::predictLeaders(Slot& slot, Value const& previousValue, - uint32_t rounds) -{ - NominationProtocol preview(slot); - preview.mPreviousValue = previousValue; - for (uint32_t i = 0; i < rounds; ++i) - { - ++preview.mRoundNumber; - preview.updateRoundLeaders(); - } - return preview.mRoundLeaders; -} - -bool -NominationProtocol::isNewerStatement(NodeID const& nodeID, - SCPNomination const& st) -{ - auto oldp = mLatestNominations.find(nodeID); - bool res = false; - - if (oldp == mLatestNominations.end()) - { - res = true; - } - else - { - res = isNewerStatement(oldp->second->getStatement().pledges.nominate(), - st); - } - return res; -} - -bool -NominationProtocol::isSubsetHelper(xdr::xvector const& p, - xdr::xvector const& v, bool& notEqual) -{ - bool res; - if (p.size() <= v.size()) - { - res = std::includes(v.begin(), v.end(), p.begin(), p.end()); - if (res) - { - notEqual = p.size() != v.size(); - } - else - { - notEqual = true; - } - } - else - { - notEqual = true; - res = false; - } - return res; -} - -SCPDriver::ValidationLevel -NominationProtocol::validateValue(Value const& v) -{ - ZoneScoped; - return mSlot.getSCPDriver().validateValue(mSlot.getSlotIndex(), v, true); -} - -ValueWrapperPtr -NominationProtocol::extractValidValue(Value const& value) -{ - ZoneScoped; - return mSlot.getSCPDriver().extractValidValue(mSlot.getSlotIndex(), value); -} - -bool -NominationProtocol::isNewerStatement(SCPNomination const& oldst, - SCPNomination const& st) -{ - bool res = false; - bool grows; - bool g = false; - - if (isSubsetHelper(oldst.votes, st.votes, g)) - { - grows = g; - if (isSubsetHelper(oldst.accepted, st.accepted, g)) - { - grows = grows || g; - res = grows; // true only if one of the sets grew - } - } - - return res; -} - -bool -NominationProtocol::isSane(SCPStatement const& st) -{ - auto const& nom = st.pledges.nominate(); - bool res = (nom.votes.size() + nom.accepted.size()) != 0; - - res = res && (std::adjacent_find( - nom.votes.begin(), nom.votes.end(), - [](stellar::Value const& l, stellar::Value const& r) { - return !(l < r); - }) == nom.votes.end()); - res = res && (std::adjacent_find( - nom.accepted.begin(), nom.accepted.end(), - [](stellar::Value const& l, stellar::Value const& r) { - return !(l < r); - }) == nom.accepted.end()); - - return res; -} - -// only called after a call to isNewerStatement so safe to replace the -// mLatestNomination -void -NominationProtocol::recordEnvelope(SCPEnvelopeWrapperPtr env) -{ - auto const& st = env->getStatement(); - auto oldp = mLatestNominations.find(st.nodeID); - if (oldp == mLatestNominations.end()) - { - mLatestNominations.insert(std::make_pair(st.nodeID, env)); - } - else - { - oldp->second = env; - } - mSlot.recordStatement(env->getStatement()); -} - -void -NominationProtocol::emitNomination() -{ - ZoneScoped; - SCPStatement st; - st.nodeID = mSlot.getLocalNode()->getNodeID(); - st.pledges.type(SCP_ST_NOMINATE); - auto& nom = st.pledges.nominate(); - - nom.quorumSetHash = mSlot.getLocalNode()->getQuorumSetHash(); - - for (auto const& v : mVotes) - { - nom.votes.emplace_back(v->getValue()); - } - for (auto const& a : mAccepted) - { - nom.accepted.emplace_back(a->getValue()); - } - - SCPEnvelope envelope = mSlot.createEnvelope(st); - - auto envW = mSlot.getSCPDriver().wrapEnvelope(envelope); - - if (mSlot.processEnvelope(envW, true) == SCP::EnvelopeState::VALID) - { - if (!mLastEnvelope || - isNewerStatement(mLastEnvelope->getStatement().pledges.nominate(), - st.pledges.nominate())) - { - mLastEnvelope = envW; - if (mSlot.isFullyValidated()) - { - mSlot.getSCPDriver().emitEnvelope(envelope); - } - } - } - else - { - // there is a bug in the application if it queued up - // a statement for itself that it considers invalid - throw std::runtime_error("moved to a bad state (nomination)"); - } -} - -#ifdef BUILD_TESTS -void -NominationProtocol::emitDelayedNomination() -{ - // While the deferred emit was pending, processEnvelope may already have - // emitted (and self-processed) a statement carrying the same votes and - // accepted values. Emitting it again would be rejected as not newer by - // our own processEnvelope, which emitNomination treats as a bad state. - SCPNomination nom; - for (auto const& v : mVotes) - { - nom.votes.emplace_back(v->getValue()); - } - for (auto const& a : mAccepted) - { - nom.accepted.emplace_back(a->getValue()); - } - if (!isNewerStatement(mSlot.getLocalNode()->getNodeID(), nom)) - { - CLOG_DEBUG(SCP, "NominationProtocol: delayed emit already sent"); - return; - } - emitNomination(); -} -#endif - -bool -NominationProtocol::acceptPredicate(Value const& v, SCPStatement const& st) -{ - auto const& nom = st.pledges.nominate(); - bool res; - res = (std::find(nom.accepted.begin(), nom.accepted.end(), v) != - nom.accepted.end()); - return res; -} - -void -NominationProtocol::applyAll(SCPNomination const& nom, - std::function processor) -{ - for (auto const& v : nom.votes) - { - processor(v); - } - // NB: "accepted" should be a subset of "votes", so this should no-op - for (auto const& a : nom.accepted) - { - if (std::find(nom.votes.begin(), nom.votes.end(), a) == nom.votes.end()) - { - processor(a); - } - } -} - -void -NominationProtocol::updateRoundLeaders() -{ - ZoneScoped; - SCPQuorumSet myQSet = mSlot.getLocalNode()->getQuorumSet(); - - auto localID = mSlot.getLocalNode()->getNodeID(); - normalizeQSet(myQSet, &localID); // excludes self - - // Count potential leaders: only include nodes with non-zero weight, - // as nodes with weight 0 can never be elected as round leaders. - size_t maxLeaderCount = 0; - if (mSlot.getSCPDriver().getNodeWeight(localID, myQSet, true) > 0) - { - ++maxLeaderCount; - } - LocalNode::forAllNodes(myQSet, [&](NodeID const& cur) { - if (mSlot.getSCPDriver().getNodeWeight(cur, myQSet, false) > 0) - { - ++maxLeaderCount; - } - return true; - }); - - // Cap the number of iterations to prevent infinite loops if something has - // gone wrong - int iterationsRemaining = 1000; - while (mRoundLeaders.size() < maxLeaderCount) - { - // initialize priority with value derived from self - std::set newRoundLeaders; - - newRoundLeaders.insert(localID); - uint64 topPriority = getNodePriority(localID, myQSet); - - LocalNode::forAllNodes(myQSet, [&](NodeID const& cur) { - uint64 w = getNodePriority(cur, myQSet); - if (w > topPriority) - { - topPriority = w; - newRoundLeaders.clear(); - } - if (w == topPriority && w > 0) - { - newRoundLeaders.insert(cur); - } - return true; - }); - - if (topPriority == 0) - { - // No one had priority, so all nodes would choose themselves - // resulting in a timeout. Clear newRoundLeaders, allowing the - // algorithm to fast timeout and try again. - newRoundLeaders.clear(); - } - - // expand mRoundLeaders with the newly computed leaders - auto oldSize = mRoundLeaders.size(); - mRoundLeaders.insert(newRoundLeaders.begin(), newRoundLeaders.end()); - if (oldSize != mRoundLeaders.size()) - { - if (Logging::logDebug("SCP")) - { - CLOG_DEBUG(SCP, "updateRoundLeaders: {} -> {}", oldSize, - mRoundLeaders.size()); - for (auto const& rl : mRoundLeaders) - { - CLOG_DEBUG(SCP, " leader {}", - mSlot.getSCPDriver().toShortString(rl)); - } - } - return; - } - else - { - mRoundNumber++; - CLOG_DEBUG(SCP, - "updateRoundLeaders: fast timeout (would no op) -> {}", - mRoundNumber); - } - - --iterationsRemaining; - if (iterationsRemaining <= 0) - { - throw std::runtime_error( - "updateRoundLeaders appears to be stuck in an infinite loop"); - } - } - CLOG_DEBUG(SCP, "updateRoundLeaders: nothing to do"); -} - -uint64 -NominationProtocol::hashNode(bool isPriority, NodeID const& nodeID) -{ - ZoneScoped; - dbgAssert(!mPreviousValue.empty()); - return mSlot.getSCPDriver().computeHashNode( - mSlot.getSlotIndex(), mPreviousValue, isPriority, mRoundNumber, nodeID); -} - -uint64 -NominationProtocol::hashValue(Value const& value) -{ - ZoneScoped; - dbgAssert(!mPreviousValue.empty()); - return mSlot.getSCPDriver().computeValueHash( - mSlot.getSlotIndex(), mPreviousValue, mRoundNumber, value); -} - -uint64 -NominationProtocol::getNodePriority(NodeID const& nodeID, - SCPQuorumSet const& qset) -{ - ZoneScoped; - uint64 res; - uint64 w = mSlot.getSCPDriver().getNodeWeight( - nodeID, qset, nodeID == mSlot.getLocalNode()->getNodeID()); - - // if w > 0; w is inclusive here as - // 0 <= hashNode <= UINT64_MAX - if (w > 0 && hashNode(false, nodeID) <= w) - { - res = hashNode(true, nodeID); - } - else - { - res = 0; - } - return res; -} - -ValueWrapperPtr -NominationProtocol::getNewValueFromNomination(SCPNomination const& nom) -{ - ZoneScoped; - // pick the highest value we don't have from the leader - // sorted using hashValue. - ValueWrapperPtr newVote; - uint64 newHash = 0; - bool foundValidValue = false; - - auto pickValue = [&](Value const& value) { - ValueWrapperPtr valueToNominate; - auto vl = validateValue(value); - if (vl >= SCPDriver::kStructurallyValidValue) - { - valueToNominate = mSlot.getSCPDriver().wrapValue(value); - } - else - { - valueToNominate = extractValidValue(value); - } - if (valueToNominate) - { - foundValidValue = true; - if (mVotes.find(valueToNominate) == mVotes.end()) - { - uint64 curHash = hashValue(valueToNominate->getValue()); - if (curHash >= newHash) - { - newHash = curHash; - newVote = valueToNominate; - } - } - } - }; - - for (auto const& val : nom.accepted) - { - pickValue(val); - } - - // Move on to votes if we have not found a valid accepted value - if (!foundValidValue) - { - for (auto const& val : nom.votes) - { - pickValue(val); - } - } - - return newVote; -} - -SCP::EnvelopeState -NominationProtocol::processEnvelope(SCPEnvelopeWrapperPtr envelope) -{ - ZoneScoped; - auto const& st = envelope->getStatement(); - auto const& nom = st.pledges.nominate(); - - if (!isNewerStatement(st.nodeID, nom)) - return SCP::EnvelopeState::INVALID; - - if (!isSane(st)) - { - CLOG_TRACE(SCP, "NominationProtocol: message didn't pass sanity check"); - return SCP::EnvelopeState::INVALID; - } - - recordEnvelope(envelope); - - // Ensure node doesn't vote on future slots - if (mNominationStarted) - { - // tracks if we should emit a new nomination message - bool modified = false; - bool newCandidates = false; - - // attempts to promote some of the votes to accepted - for (auto const& v : nom.votes) - { - auto vw = mSlot.getSCPDriver().wrapValue(v); - if (mAccepted.find(vw) != mAccepted.end()) - { // v is already accepted - continue; - } - if (mSlot.federatedAccept( - [&v](SCPStatement const& st) -> bool { - auto const& nom = st.pledges.nominate(); - bool res; - res = (std::find(nom.votes.begin(), nom.votes.end(), - v) != nom.votes.end()); - return res; - }, - std::bind(&NominationProtocol::acceptPredicate, v, _1), - mLatestNominations)) - { - auto vl = validateValue(v); - if (vl >= SCPDriver::kStructurallyValidValue) - { - mAccepted.emplace(vw); - mVotes.emplace(vw); - modified = true; - } - else - { - // the value made it pretty far: - // see if we can vote for a variation that - // we consider valid - auto toVote = extractValidValue(v); - if (toVote) - { - if (mVotes.emplace(toVote).second) - { - modified = true; - } - } - } - } - } - // attempts to promote accepted values to candidates - for (auto const& a : mAccepted) - { - if (mCandidates.find(a) != mCandidates.end()) - { - continue; - } - if (mSlot.federatedRatify( - std::bind(&NominationProtocol::acceptPredicate, - a->getValue(), _1), - mLatestNominations)) - { - mCandidates.emplace(a); - newCandidates = true; - // Stop the timer, as there's no need to continue nominating, - // per the whitepaper: - // "As soon as `v` has a candidate value, however, it must cease - // voting to nominate `x` for any new values `x`" - mSlot.getSCPDriver().stopTimer(mSlot.getSlotIndex(), - Slot::NOMINATION_TIMER); - } - } - - // only take round leader votes if we're still looking for - // candidates - if (mCandidates.empty() && - mRoundLeaders.find(st.nodeID) != mRoundLeaders.end()) - { - auto newVote = getNewValueFromNomination(nom); - if (newVote) - { - mVotes.emplace(newVote); - modified = true; - mSlot.getSCPDriver().nominatingValue(mSlot.getSlotIndex(), - newVote->getValue()); - } - } - - if (modified) - { - emitNomination(); - } - - if (newCandidates) - { - mLatestCompositeCandidate = mSlot.getSCPDriver().combineCandidates( - mSlot.getSlotIndex(), mCandidates); - - mSlot.getSCPDriver().updatedCandidateValue( - mSlot.getSlotIndex(), mLatestCompositeCandidate->getValue()); - - mSlot.bumpState(mLatestCompositeCandidate->getValue(), false); - } - } - return SCP::EnvelopeState::VALID; -} - -std::vector -NominationProtocol::getStatementValues(SCPStatement const& st) -{ - std::vector res; - applyAll(st.pledges.nominate(), - [&](Value const& v) { res.emplace_back(v); }); - return res; -} - -void -NominationProtocol::stripUpgrades(ValueWrapperPtr& value) const -{ - ZoneScoped; - ValueWrapperPtr const newValue = - mSlot.getSCPDriver().stripAllUpgrades(value->getValue()); - if (newValue != nullptr && newValue->getValue() != value->getValue()) - { - CLOG_WARNING(SCP, - "Due to high timeouts, stripping upgrades from " - "nominated value: {} -> {}", - mSlot.getSCP().getValueString(value->getValue()), - mSlot.getSCP().getValueString(newValue->getValue())); - value = newValue; - } -} - -// attempts to nominate a value for consensus -bool -NominationProtocol::nominate(NominationValueSupplier const& makeValue, - Value const& previousValue, bool timedout) -{ - ZoneScoped; - - // No need to continue nominating, as per the whitepaper: - // "As soon as `v` has a candidate value, however, it must cease - // voting to nominate `x` for any new values `x`" - if (!mCandidates.empty()) - { - CLOG_DEBUG(SCP, "Skip nomination round {}, already have a candidate", - mRoundNumber); - return false; - } - - CLOG_DEBUG(SCP, "NominationProtocol::nominate ({})", mRoundNumber); - - bool updated = false; - - if (timedout) - { - mTimerExpCount++; - } - - if (timedout && !mNominationStarted) - { - CLOG_DEBUG(SCP, "NominationProtocol::nominate (TIMED OUT)"); - return false; - } - - mNominationStarted = true; - - mPreviousValue = previousValue; - - mRoundNumber++; - updateRoundLeaders(); - - std::chrono::milliseconds timeout = mSlot.getSCPDriver().computeTimeout( - mRoundNumber, /*isNomination=*/true); - - // add a few more values from other leaders - for (auto const& leader : mRoundLeaders) - { - auto it = mLatestNominations.find(leader); - if (it != mLatestNominations.end()) - { - auto lnmV = getNewValueFromNomination( - it->second->getStatement().pledges.nominate()); - if (lnmV) - { - mVotes.insert(lnmV); - updated = true; - mSlot.getSCPDriver().nominatingValue(mSlot.getSlotIndex(), - lnmV->getValue()); - } - } - } - - // Check if we are a leader for this round - if (mRoundLeaders.find(mSlot.getLocalNode()->getNodeID()) != - mRoundLeaders.end()) - { - // Check whether we've exceeded the upgrade timeout limit for this slot - bool const overUpgradeTimeoutLimit = - mTimerExpCount >= - mSlot.getSCPDriver().getUpgradeNominationTimeoutLimit(); - - bool shouldVoteForValue = false; - bool shouldStripUpgrades = false; - // Add our value if we haven't added any votes yet. - if (mVotes.empty()) - { - shouldVoteForValue = true; - } - - if (overUpgradeTimeoutLimit) - { - // We've exceeded the upgrade timeout limit. First, check whether - // all of our votes have upgrades. We only want to add a new value - // with stripped upgrades if there's a chance that it will help - // alleviate timeouts due to contentious upgrades. If we're already - // voting for a value without upgrades, then adding another is - // unlikely to help, as we are likely in one of the two following - // scenarios: - // 1. The current timeout situation is unrelated to upgrades - // 2. We just happened to add a value without upgrades earlier in - // this function - // In either case, adding a new value without upgrades will not - // help, and may make things worse by putting more strain on the - // network. - bool const allVotesHaveUpgrades = std::all_of( - mVotes.begin(), mVotes.end(), [&](ValueWrapperPtr const& v) { - return mSlot.getSCPDriver().hasUpgrades(v->getValue()); - }); - - if (allVotesHaveUpgrades) - { - // All votes have upgrades, so strip upgrades from `value` and - // vote for it. - shouldStripUpgrades = true; - shouldVoteForValue = true; - } - } - - if (shouldVoteForValue) - { - auto value = makeValue(); - // Making the value can make pending envelopes ready and reenter - // SCP, potentially externalizing this slot or creating a candidate. - // Do not vote or rearm its timer after nomination has stopped. - if (!mNominationStarted || !mCandidates.empty()) - { - return false; - } - if (value) - { - if (shouldStripUpgrades) - { - stripUpgrades(value); - } - auto ins = mVotes.insert(value); - if (ins.second) - { - updated = true; - mSlot.getSCPDriver().nominatingValue(mSlot.getSlotIndex(), - value->getValue()); - } - } - } - } - - std::shared_ptr slot = mSlot.shared_from_this(); - mSlot.getSCPDriver().setupTimer( - mSlot.getSlotIndex(), Slot::NOMINATION_TIMER, timeout, - [slot, makeValue, previousValue]() { - slot->nominate(makeValue, previousValue, true); - }); - -#ifdef BUILD_TESTS - // If a nomination-emit delay is configured, defer the emit by - // arming NOMINATION_EMIT_TIMER instead of broadcasting now. A subsequent - // nominate() will replace the timer so the eventual emit reflects the - // latest state. Skip the deferral once the SCP nomination timeout has - // fired (`timedout`) so retries don't compound latency, and cancel any - // still-armed timer in that case. - if (updated) - { - auto delay = mSlot.getSCPDriver().getNominationEmitDelayForTesting(); - if (delay > std::chrono::milliseconds::zero() && !timedout) - { - mSlot.getSCPDriver().setupTimer( - mSlot.getSlotIndex(), Slot::NOMINATION_EMIT_TIMER, delay, - [slot, this]() { emitDelayedNomination(); }); - return updated; - } - if (delay > std::chrono::milliseconds::zero()) - { - mSlot.getSCPDriver().stopTimer(mSlot.getSlotIndex(), - Slot::NOMINATION_EMIT_TIMER); - } - } -#endif - - if (updated) - { - emitNomination(); - } - else - { - CLOG_DEBUG(SCP, "NominationProtocol::nominate (SKIPPED)"); - } - - return updated; -} - -void -NominationProtocol::stopNomination() -{ - mNominationStarted = false; -} - -std::set const& -NominationProtocol::getLeaders() const -{ - return mRoundLeaders; -} - -Json::Value -NominationProtocol::getJsonInfo() -{ - Json::Value ret; - ret["roundnumber"] = mRoundNumber; - ret["started"] = mNominationStarted; - - int counter = 0; - for (auto const& v : mVotes) - { - ret["X"][counter] = mSlot.getSCP().getValueString(v->getValue()); - counter++; - } - - counter = 0; - for (auto const& v : mAccepted) - { - ret["Y"][counter] = mSlot.getSCP().getValueString(v->getValue()); - counter++; - } - - counter = 0; - for (auto const& v : mCandidates) - { - ret["Z"][counter] = mSlot.getSCP().getValueString(v->getValue()); - counter++; - } - - return ret; -} - -SCP::QuorumInfoNodeState -NominationProtocol::getState(NodeID const& n, bool selfAlreadyMovedOn) -{ - auto state = SCP::QuorumInfoNodeState::AGREE; - if (n == mSlot.getLocalNode()->getNodeID()) - { - // always mark myself as AGREE. - return state; - } - - bool enoughTimeHasPassed = - mTimerExpCount >= Slot::NUM_TIMEOUTS_THRESHOLD_FOR_REPORTING || - - selfAlreadyMovedOn; - - auto it = mLatestNominations.find(n); - if (it == mLatestNominations.end()) - { - if (enoughTimeHasPassed) - { - state = SCP::QuorumInfoNodeState::MISSING; - } - else - { - // Not enough time has passed to start calling this node missing. - state = SCP::QuorumInfoNodeState::NO_INFO; - } - } - else - { - if (enoughTimeHasPassed && mLastEnvelope) - { - // Enough time has passed & we have some information on this node. - // We'll examine the information to determine the state of this - // node. - auto const& other = - it->second->getStatement().pledges.nominate().accepted; - auto const& mine = - mLastEnvelope->getStatement().pledges.nominate().accepted; - - bool notEq, isSubset = isSubsetHelper(other, mine, notEq); - if (isSubset) - { - if (notEq) - { - // If there's some values that I've accepted that - // they haven't, they are slow. - state = SCP::QuorumInfoNodeState::DELAYED; - } - } - else - { - // If they have accepted a value that I haven't, - // it's likely that I'll never accept it. - state = SCP::QuorumInfoNodeState::DISAGREE; - } - } - } - return state; -} - -void -NominationProtocol::setStateFromEnvelope(SCPEnvelopeWrapperPtr e) -{ - if (mNominationStarted) - { - throw std::runtime_error( - "Cannot set state after nomination is started"); - } - recordEnvelope(e); - auto const& nom = e->getStatement().pledges.nominate(); - for (auto const& a : nom.accepted) - { - mAccepted.emplace(mSlot.getSCPDriver().wrapValue(a)); - } - for (auto const& v : nom.votes) - { - mVotes.emplace(mSlot.getSCPDriver().wrapValue(v)); - } - - mLastEnvelope = e; -} - -bool -NominationProtocol::processCurrentState( - std::function const& f, bool forceSelf) const -{ - for (auto const& n : mLatestNominations) - { - // only return messages for self if the slot is fully validated - if (forceSelf || !(n.first == mSlot.getSCP().getLocalNodeID()) || - mSlot.isFullyValidated()) - { - if (!f(n.second->getEnvelope())) - { - return false; - } - } - } - return true; -} - -SCPEnvelope const* -NominationProtocol::getLatestMessage(NodeID const& id) const -{ - auto it = mLatestNominations.find(id); - if (it != mLatestNominations.end()) - { - return &it->second->getEnvelope(); - } - return nullptr; -} -} diff --git a/src/scp/NominationProtocol.h b/src/scp/NominationProtocol.h deleted file mode 100644 index cf0e3656e1..0000000000 --- a/src/scp/NominationProtocol.h +++ /dev/null @@ -1,152 +0,0 @@ -// Copyright 2014 Stellar Development Foundation and contributors. Licensed -// under the Apache License, Version 2.0. See the COPYING file at the root -// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 - -#pragma once - -#include "lib/json/json-forwards.h" -#include "scp/SCP.h" -#include -#include -#include -#include -#include - -namespace stellar -{ -class NominationProtocol -{ - protected: - Slot& mSlot; - - int32 mRoundNumber; - - ValueWrapperPtrSet mVotes; // X - ValueWrapperPtrSet mAccepted; // Y - ValueWrapperPtrSet mCandidates; // Z - std::map mLatestNominations; // N - - SCPEnvelopeWrapperPtr mLastEnvelope; // last envelope emitted by this node - - // nodes from quorum set that have the highest priority this round - std::set mRoundLeaders; - - // true if 'nominate' was called - bool mNominationStarted; - - // the latest (if any) candidate value - ValueWrapperPtr mLatestCompositeCandidate; - - // the value from the previous slot - Value mPreviousValue; - - bool isNewerStatement(NodeID const& nodeID, SCPNomination const& st); - - // returns true if 'p' is a subset of 'v' - // also sets 'notEqual' if p and v differ - // note: p and v must be sorted - static bool isSubsetHelper(xdr::xvector const& p, - xdr::xvector const& v, bool& notEqual); - - SCPDriver::ValidationLevel validateValue(Value const& v); - ValueWrapperPtr extractValidValue(Value const& value); - - bool isSane(SCPStatement const& st); - - void recordEnvelope(SCPEnvelopeWrapperPtr env); - - void emitNomination(); - -#ifdef BUILD_TESTS - // Fires the deferred emit armed by nominate() when a nomination-emit - // delay is configured. Skips the emit if this node already sent a - // statement at least as recent in the meantime. - void emitDelayedNomination(); -#endif - - // returns true if v is in the accepted list from the statement - static bool acceptPredicate(Value const& v, SCPStatement const& st); - - // applies 'processor' to all values from the passed in nomination - static void applyAll(SCPNomination const& nom, - std::function processor); - - // updates the set of nodes that have priority over the others - void updateRoundLeaders(); - - // computes Gi(isPriority?P:N, prevValue, mRoundNumber, nodeID) - // from the paper - uint64 hashNode(bool isPriority, NodeID const& nodeID); - - // computes Gi(K, prevValue, mRoundNumber, value) - uint64 hashValue(Value const& value); - - uint64 getNodePriority(NodeID const& nodeID, SCPQuorumSet const& qset); - - // returns the highest value that we don't have yet, that we should - // vote for, extracted from a nomination. - // returns nullptr if no new value was found - ValueWrapperPtr getNewValueFromNomination(SCPNomination const& nom); - - public: - static bool isNewerStatement(SCPNomination const& oldst, - SCPNomination const& st); - - NominationProtocol(Slot& slot); - - // Preview the cumulative leaders for the first `rounds` nomination calls. - // Uses the same fast-forward/tie handling as live nomination, without - // starting nomination, arming timers, or emitting statements. - static std::set - predictLeaders(Slot& slot, Value const& previousValue, uint32_t rounds); - - SCP::EnvelopeState processEnvelope(SCPEnvelopeWrapperPtr envelope); - - static std::vector getStatementValues(SCPStatement const& st); - - // attempts to nominate a value for consensus - bool nominate(NominationValueSupplier const& makeValue, - Value const& previousValue, bool timedout); - - // stops the nomination protocol - void stopNomination(); - - // return the current leaders - std::set const& getLeaders() const; - - ValueWrapperPtr const& - getLatestCompositeCandidate() const - { - return mLatestCompositeCandidate; - } - - Json::Value getJsonInfo(); - - SCP::QuorumInfoNodeState getState(NodeID const& node, - bool selfAlreadyMovedOn); - - SCPEnvelope const* - getLastMessageSend() const - { - return mLastEnvelope ? &mLastEnvelope->getEnvelope() : nullptr; - } - - void setStateFromEnvelope(SCPEnvelopeWrapperPtr e); - - bool processCurrentState(std::function const& f, - bool forceSelf) const; - - // returns the latest message from a node - // or nullptr if not found - SCPEnvelope const* getLatestMessage(NodeID const& id) const; - - private: - // The number of times the timer has expired - // Used for the quorum endpoint. - uint32_t mTimerExpCount; - - // Strip any upgrades that `value` may have, modifying it in place. Does - // nothing if `value` has no upgrades. - void stripUpgrades(ValueWrapperPtr& value) const; -}; -} diff --git a/src/scp/SCP.cpp b/src/scp/SCP.cpp index 8afbc3a6b3..f8f5641f7c 100644 --- a/src/scp/SCP.cpp +++ b/src/scp/SCP.cpp @@ -34,29 +34,17 @@ SCP::receiveEnvelope(SCPEnvelopeWrapperPtr envelope) } bool -SCP::nominate(uint64 slotIndex, NominationValueSupplier const& makeValue, - Value const& previousValue) +SCP::startSlot(uint64 slotIndex, ProposalValueSupplier const& makeValue) { dbgAssert(isValidator()); - return getSlot(slotIndex, true)->nominate(makeValue, previousValue, false); + return getSlot(slotIndex, true)->startLeaderRounds(makeValue); } -std::set -SCP::predictNominationLeaders(uint64 slotIndex, Value const& previousValue, - uint32_t rounds) -{ - Slot preview(slotIndex, *this); - return NominationProtocol::predictLeaders(preview, previousValue, rounds); -} - -void -SCP::stopNomination(uint64 slotIndex) +uint32 +SCP::getLeaderRound(uint64 slotIndex) { - auto s = getSlot(slotIndex, false); - if (s) - { - s->stopNomination(); - } + auto slot = getSlot(slotIndex, false); + return slot ? slot->getLeaderRound() : 0; } void @@ -287,17 +275,6 @@ SCP::getMissingNodes(NodeID const& id, uint64 index) return ret; } -std::set -SCP::getNominationLeaders(uint64 slotIndex) -{ - auto slot = getSlot(slotIndex, false); - if (slot) - { - return slot->getNominationLeaders(); - } - return {}; -} - bool SCP::isValidator() { @@ -433,8 +410,7 @@ SCP::getLatestMessage(NodeID const& id) } bool -SCP::isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt) +SCP::isNewerStatement(SCPStatement const& oldSt, SCPStatement const& newSt) { if (oldSt.slotIndex != newSt.slotIndex || !(oldSt.nodeID == newSt.nodeID)) { @@ -444,7 +420,7 @@ SCP::isNewerNominationOrBallotSt(SCPStatement const& oldSt, auto slot = getSlot(oldSt.slotIndex, false); if (slot) { - return slot->isNewerNominationOrBallotSt(oldSt, newSt); + return slot->isNewerStatement(oldSt, newSt); } return false; diff --git a/src/scp/SCP.h b/src/scp/SCP.h index e49e3535de..2fe4fbd7ba 100644 --- a/src/scp/SCP.h +++ b/src/scp/SCP.h @@ -51,13 +51,16 @@ class SCP // invokes the appropriate methods EnvelopeState receiveEnvelope(SCPEnvelopeWrapperPtr envelope); - // Start nomination for slotIndex, constructing a local value only if needed - // previousValue is the value from slotIndex-1 - bool nominate(uint64 slotIndex, NominationValueSupplier const& makeValue, - Value const& previousValue); + // Starts consensus on slotIndex for the local node (its trigger fired). + // There is no nomination phase: each round (ballot counter) has one leader + // given by SCPDriver::getRoundLeader, and nodes vote for the leader's + // ballot. `makeValue` is invoked only if the local node becomes the leader + // of a round and has no value to put forward. + bool startSlot(uint64 slotIndex, ProposalValueSupplier const& makeValue); - // stops nomination for a slot - void stopNomination(uint64 slotIndex); + // The round the local node is in for slotIndex (0 if it has not started + // the slot or does not know it). + uint32 getLeaderRound(uint64 slotIndex); // Local QuorumSet interface (can be dynamically updated) void updateLocalQuorumSet(SCPQuorumSet const& qSet); @@ -90,14 +93,6 @@ class SCP // Get a list of nodes from id's quorum set that are missing in consensus std::set getMissingNodes(NodeID const& id, uint64 index = 0); - // returns the current nomination leaders for the given slot - std::set getNominationLeaders(uint64 slotIndex); - - // A read-only preview; does not create or modify a live SCP slot. - std::set predictNominationLeaders(uint64 slotIndex, - Value const& previousValue, - uint32_t rounds); - // Purges all data relative to slots that fall outside the range // [minSlotIndex, maxSlotIndex]. Either bound may be nullopt to skip // that direction. The slotToKeep slot is never purged. @@ -148,8 +143,8 @@ class SCP // or nullptr if not found SCPEnvelope const* getLatestMessage(NodeID const& id); - bool isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt); + // returns true if newSt is newer than oldSt (same node, same slot) + bool isNewerStatement(SCPStatement const& oldSt, SCPStatement const& newSt); // returns messages that contributed to externalizing the slot // (or empty if the slot didn't externalize) diff --git a/src/scp/SCPDriver.cpp b/src/scp/SCPDriver.cpp index 9e54c07778..4f4137e518 100644 --- a/src/scp/SCPDriver.cpp +++ b/src/scp/SCPDriver.cpp @@ -10,28 +10,11 @@ #include "crypto/KeyUtils.h" #include "crypto/SecretKey.h" #include "util/GlobalChecks.h" -#include "util/numeric.h" #include "xdrpp/marshal.h" namespace stellar { -namespace -{ -uint64 -computeWeight(uint64 m, uint64 total, uint64 threshold) -{ - uint64 res; - releaseAssert(threshold <= total); - // Since threshold <= total, calculating res=m*threshold/total will always - // produce res <= m, and we do not need to handle the possibility of this - // call returning false (indicating overflow). - bool noOverflow = bigDivideUnsigned(res, m, threshold, total, ROUND_UP); - releaseAssert(noOverflow); - return res; -} -} // namespace - bool WrappedValuePtrComparator::operator()(ValueWrapperPtr const& l, ValueWrapperPtr const& r) const @@ -90,95 +73,4 @@ SCPDriver::toShortString(NodeID const& pk) const return KeyUtils::toShortString(pk); } -// values used to switch hash function between priority and neighborhood checks -static uint32 const hash_N = 1; -static uint32 const hash_P = 2; -static uint32 const hash_K = 3; - -uint64 -SCPDriver::hashHelper( - uint64 slotIndex, Value const& prev, - std::function>&)> extra) -{ - std::vector> vals; - vals.emplace_back(xdr::xdr_to_opaque(slotIndex)); - vals.emplace_back(xdr::xdr_to_opaque(prev)); - extra(vals); - Hash t = getHashOf(vals); - uint64 res = 0; - for (size_t i = 0; i < sizeof(res); i++) - { - res = (res << 8) | t[i]; - } - return res; -} - -uint64 -SCPDriver::computeHashNode(uint64 slotIndex, Value const& prev, bool isPriority, - int32_t roundNumber, NodeID const& nodeID) -{ -#ifdef BUILD_TESTS - if (mPriorityLookupForTesting) - { - return isPriority ? mPriorityLookupForTesting(nodeID) : 0; - } -#endif - return hashHelper( - slotIndex, prev, [&](std::vector>& vals) { - vals.emplace_back(xdr::xdr_to_opaque(isPriority ? hash_P : hash_N)); - vals.emplace_back(xdr::xdr_to_opaque(roundNumber)); - vals.emplace_back(xdr::xdr_to_opaque(nodeID)); - }); -} - -uint64 -SCPDriver::computeValueHash(uint64 slotIndex, Value const& prev, - int32_t roundNumber, Value const& value) -{ - return hashHelper(slotIndex, prev, - [&](std::vector>& vals) { - vals.emplace_back(xdr::xdr_to_opaque(hash_K)); - vals.emplace_back(xdr::xdr_to_opaque(roundNumber)); - vals.emplace_back(xdr::xdr_to_opaque(value)); - }); -} - -// if a validator is repeated multiple times its weight is only the -// weight of the first occurrence -uint64 -SCPDriver::getNodeWeight(NodeID const& nodeID, SCPQuorumSet const& qset, - bool isLocalNode) const -{ - if (isLocalNode) - { - // local node is in all quorum sets - return UINT64_MAX; - } - - uint64 n = qset.threshold; - uint64 d = qset.innerSets.size() + qset.validators.size(); - uint64 res; - - for (auto const& qsetNode : qset.validators) - { - if (qsetNode == nodeID) - { - res = computeWeight(UINT64_MAX, d, n); - return res; - } - } - - for (auto const& q : qset.innerSets) - { - uint64 leafW = SCPDriver::getNodeWeight(nodeID, q, isLocalNode); - if (leafW) - { - res = computeWeight(leafW, d, n); - return res; - } - } - - return 0; -} - } diff --git a/src/scp/SCPDriver.h b/src/scp/SCPDriver.h index 19fcbfc5ec..a41af7d781 100644 --- a/src/scp/SCPDriver.h +++ b/src/scp/SCPDriver.h @@ -44,10 +44,10 @@ class ValueWrapper : public NonMovableOrCopyable typedef std::shared_ptr SCPQuorumSetPtr; typedef std::shared_ptr ValueWrapperPtr; -// Called only when the local node is a nomination leader and needs its own -// value. May return nullptr if the application can no longer propose for the -// slot. The callback must remain valid across nomination timeouts. -using NominationValueSupplier = std::function; +// Called only when the local node is the leader of a round and needs to build +// its own value. May return nullptr if the application can no longer propose +// for the slot. The callback must remain valid for the lifetime of the slot. +using ProposalValueSupplier = std::function; class WrappedValuePtrComparator { @@ -155,7 +155,10 @@ class SCPDriver // be validated due to parallel downloading (e.g. it's for LCL+1 but the // node is still downloading the tx set), then kStructurallyValidValue // should be used. - // validation can be *more* restrictive during nomination as needed + // validation is *more* restrictive when `proposal` is set: that mode is + // used before the local node adopts (votes for) a round leader's value, + // and for the leader's own value. It is where the application enforces + // what it is willing to propose (e.g. only upgrades it has armed). // NB: validation levels are ordered enum ValidationLevel { @@ -172,22 +175,11 @@ class SCPDriver kFullyValidatedValue = 3 }; virtual ValidationLevel - validateValue(uint64 slotIndex, Value const& value, bool nomination) const + validateValue(uint64 slotIndex, Value const& value, bool proposal) const { return kMaybeValidNotCurrentValue; } - // `extractValidValue` transforms the value, if possible to a different - // value that the local node would agree to (fully validated). - // This is used during nomination when encountering an invalid value (ie - // validateValue did not return `kFullyValidatedValue` for this value). - // returning nullptr means no valid value could be extracted - virtual ValueWrapperPtr - extractValidValue(uint64 slotIndex, Value const& value) - { - return nullptr; - } - // Helper function to craft an empty-tx-set value from a Value. virtual Value makeEmptyTxSetValueFromValue(Value const& v) const = 0; @@ -215,33 +207,23 @@ class SCPDriver virtual Hash getHashOf(std::vector> const& vals) const = 0; - // `computeHashNode` is used by the nomination protocol to - // randomize the order of messages between nodes. - virtual uint64 computeHashNode(uint64 slotIndex, Value const& prev, - bool isPriority, int32_t roundNumber, - NodeID const& nodeID); - - // `computeValueHash` is used by the nomination protocol to - // randomize the relative order between values. - virtual uint64 computeValueHash(uint64 slotIndex, Value const& prev, - int32_t roundNumber, Value const& value); - - // `combineCandidates` computes the composite value based off a list - // of candidate values. - virtual ValueWrapperPtr - combineCandidates(uint64 slotIndex, - ValueWrapperPtrSet const& candidates) = 0; - - // Checks whether `v` contains upgrades - virtual bool hasUpgrades(Value const& v) = 0; - - // `stripAllUpgrades` returns a new value with all upgrades removed - virtual ValueWrapperPtr stripAllUpgrades(Value const& v) = 0; - - // Returns the maximum number of nomination timeouts permitted per slot for - // the currently set upgrade. Defaults to the maximum uint32_t value - // (effectively unlimited) if the upgrade does not specify a limit. - virtual uint32_t getUpgradeNominationTimeoutLimit() const = 0; + // `getRoundLeader` returns the leader of round `round` (1-based, a round + // is a ballot counter) for `slotIndex`. Every node that takes part in + // consensus on the slot must compute the same answer. Returns nullopt when + // the schedule is not known for that slot (e.g. the slot is not the next + // one to close), in which case the local node neither proposes nor adopts. + virtual std::optional getRoundLeader(uint64 slotIndex, + uint32 round) = 0; + + // `isProposalPending` returns true if the application holds a statement + // of `leader` for `slotIndex` that it has not handed to SCP yet (e.g. it + // is still fetching the transaction set of its value). SCP then gives a + // round leader one more timeout before moving to the next round. + virtual bool + isProposalPending(uint64 slotIndex, NodeID const& leader) const + { + return false; + } // `setupTimer`: requests to trigger 'cb' after timeout // if cb is nullptr, the timer is cancelled @@ -253,22 +235,7 @@ class SCPDriver // `computeTimeout` computes a timeout given a round number // it should be sufficiently large such that nodes in a // quorum can exchange 4 messages - virtual std::chrono::milliseconds computeTimeout(uint32 roundNumber, - bool isNomination) = 0; - -#ifdef BUILD_TESTS - virtual std::chrono::milliseconds - getNominationEmitDelayForTesting() const - { - return std::chrono::milliseconds::zero(); - } -#endif - - // returns the weight of the node within the qset normalized between - // 0-UINT64_MAX. If `nodeID` is the local node, then set `isLocalNode` to - // `true`. - virtual uint64 getNodeWeight(NodeID const& nodeID, SCPQuorumSet const& qset, - bool isLocalNode) const; + virtual std::chrono::milliseconds computeTimeout(uint32 roundNumber) = 0; // Inform about events happening within the consensus algorithm. @@ -284,21 +251,20 @@ class SCPDriver { } - // ``nominatingValue`` is called every time the local instance nominates - // a new value. + // the following methods are used for monitoring of the SCP subsystem + // most implementation don't really need to do anything with these + + // `proposedValue` is called when the local node, as the leader of + // `round`, puts `value` forward. virtual void - nominatingValue(uint64 slotIndex, Value const& value) + proposedValue(uint64 slotIndex, uint32 round, Value const& value) { } - // the following methods are used for monitoring of the SCP subsystem - // most implementation don't really need to do anything with these - - // `updatedCandidateValue` is called every time a new candidate value - // is included in the candidate set, the value passed in is - // a composite value + // `enteredRound` is called when the local node moves to `round` for + // `slotIndex`. `timedOut` is true when a timer expiry caused the move. virtual void - updatedCandidateValue(uint64 slotIndex, Value const& value) + enteredRound(uint64 slotIndex, uint32 round, bool timedOut) { } @@ -347,19 +313,5 @@ class SCPDriver measureAndRecordBallotBlockedOnTxSet(uint64 slotIndex, Value const& value) { } - -#ifdef BUILD_TESTS - std::function mPriorityLookupForTesting; - void - setPriorityLookup(std::function const& f) - { - mPriorityLookupForTesting = f; - } -#endif - - private: - uint64 - hashHelper(uint64 slotIndex, Value const& prev, - std::function>&)> extra); }; } diff --git a/src/scp/Slot.cpp b/src/scp/Slot.cpp index 4971fc2e27..37ce296ef7 100644 --- a/src/scp/Slot.cpp +++ b/src/scp/Slot.cpp @@ -24,31 +24,18 @@ Slot::Slot(uint64 slotIndex, SCP& scp) : mSlotIndex(slotIndex) , mSCP(scp) , mBallotProtocol(*this) - , mNominationProtocol(*this) , mFullyValidated(scp.getLocalNode()->isValidator()) , mGotVBlocking(false) { } -ValueWrapperPtr const& -Slot::getLatestCompositeCandidate() -{ - return mNominationProtocol.getLatestCompositeCandidate(); -} - std::vector Slot::getLatestMessagesSend() const { std::vector res; if (mFullyValidated) { - SCPEnvelope const* e; - e = mNominationProtocol.getLastMessageSend(); - if (e) - { - res.emplace_back(*e); - } - e = mBallotProtocol.getLastMessageSend(); + SCPEnvelope const* e = mBallotProtocol.getLastMessageSend(); if (e) { res.emplace_back(*e); @@ -68,12 +55,11 @@ Slot::setStateFromEnvelope(SCPEnvelopeWrapperPtr env) if (e.statement.pledges.type() == SCPStatementType::SCP_ST_NOMINATE) { - mNominationProtocol.setStateFromEnvelope(env); - } - else - { - mBallotProtocol.setStateFromEnvelope(env); + // Nomination no longer exists: ignore state saved by a build that + // still had it. + return; } + mBallotProtocol.setStateFromEnvelope(env); if (!prev) { @@ -91,53 +77,24 @@ void Slot::processCurrentState(std::function const& f, bool forceSelf) const { - mNominationProtocol.processCurrentState(f, forceSelf) && - mBallotProtocol.processCurrentState(f, forceSelf); + mBallotProtocol.processCurrentState(f, forceSelf); } SCPEnvelope const* Slot::getLatestMessage(NodeID const& id) const { - auto m = mBallotProtocol.getLatestMessage(id); - if (m == nullptr) - { - m = mNominationProtocol.getLatestMessage(id); - } - return m; + return mBallotProtocol.getLatestMessage(id); } bool -Slot::isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt) +Slot::isNewerStatement(SCPStatement const& oldSt, SCPStatement const& newSt) { - bool oldNomination = - oldSt.pledges.type() == SCPStatementType::SCP_ST_NOMINATE; - bool newNomination = - newSt.pledges.type() == SCPStatementType::SCP_ST_NOMINATE; - - if (oldNomination != newNomination) + if (oldSt.pledges.type() == SCPStatementType::SCP_ST_NOMINATE || + newSt.pledges.type() == SCPStatementType::SCP_ST_NOMINATE) { return false; } - - bool replace = false; - if (oldNomination) - { - if (NominationProtocol::isNewerStatement(oldSt.pledges.nominate(), - newSt.pledges.nominate())) - { - replace = true; - } - } - else - { - if (BallotProtocol::isNewerStatement(oldSt, newSt)) - { - replace = true; - } - } - - return replace; + return BallotProtocol::isNewerStatement(oldSt, newSt); } std::vector @@ -173,12 +130,11 @@ Slot::processEnvelope(SCPEnvelopeWrapperPtr envelope, bool self) if (st.pledges.type() == SCPStatementType::SCP_ST_NOMINATE) { - res = mNominationProtocol.processEnvelope(envelope); - } - else - { - res = mBallotProtocol.processEnvelope(envelope, self); + // There is no nomination phase: the round leader's ballot is the + // proposal. Nomination statements are dropped. + return SCP::EnvelopeState::INVALID; } + res = mBallotProtocol.processEnvelope(envelope, self); if (!prev && res == SCP::VALID) { @@ -211,12 +167,6 @@ Slot::processEnvelope(SCPEnvelopeWrapperPtr envelope, bool self) return res; } -bool -Slot::abandonBallot() -{ - return mBallotProtocol.abandonBallot(0); -} - bool Slot::bumpState(Value const& value, bool force) { @@ -225,22 +175,15 @@ Slot::bumpState(Value const& value, bool force) } bool -Slot::nominate(NominationValueSupplier const& makeValue, - Value const& previousValue, bool timedout) +Slot::startLeaderRounds(ProposalValueSupplier const& makeValue) { - return mNominationProtocol.nominate(makeValue, previousValue, timedout); + return mBallotProtocol.startLeaderRounds(makeValue); } -void -Slot::stopNomination() +uint32 +Slot::getLeaderRound() const { - mNominationProtocol.stopNomination(); -} - -std::set -Slot::getNominationLeaders() const -{ - return mNominationProtocol.getLeaders(); + return mBallotProtocol.getLeaderRound(); } bool @@ -300,7 +243,9 @@ Slot::getStatementValues(SCPStatement const& st) std::vector res; if (st.pledges.type() == SCP_ST_NOMINATE) { - res = NominationProtocol::getStatementValues(st); + auto const& nom = st.pledges.nominate(); + res.insert(res.end(), nom.votes.begin(), nom.votes.end()); + res.insert(res.end(), nom.accepted.begin(), nom.accepted.end()); } else { @@ -378,7 +323,6 @@ Slot::getJsonInfo(bool fullKeys) } ret["validated"] = mFullyValidated; - ret["nomination"] = mNominationProtocol.getJsonInfo(); ret["ballotProtocol"] = mBallotProtocol.getJsonInfo(); return ret; @@ -387,12 +331,7 @@ Slot::getJsonInfo(bool fullKeys) SCP::QuorumInfoNodeState Slot::getState(NodeID const& node, bool selfAlreadyMovedOn) { - auto b = mBallotProtocol.getState(node, selfAlreadyMovedOn); - if (b != SCP::QuorumInfoNodeState::NO_INFO) - { - return b; - } - return mNominationProtocol.getState(node, selfAlreadyMovedOn); + return mBallotProtocol.getState(node, selfAlreadyMovedOn); } Json::Value diff --git a/src/scp/Slot.h b/src/scp/Slot.h index 3b556a904e..06c6dbfb9b 100644 --- a/src/scp/Slot.h +++ b/src/scp/Slot.h @@ -6,7 +6,6 @@ #include "BallotProtocol.h" #include "LocalNode.h" -#include "NominationProtocol.h" #include "lib/json/json-forwards.h" #include "scp/SCP.h" #include @@ -29,7 +28,6 @@ class Slot : public std::enable_shared_from_this SCP& mSCP; BallotProtocol mBallotProtocol; - NominationProtocol mNominationProtocol; // keeps track of all statements seen so far for this slot. // it is used for debugging purpose @@ -81,8 +79,6 @@ class Slot : public std::enable_shared_from_this return mBallotProtocol; } - ValueWrapperPtr const& getLatestCompositeCandidate(); - // returns the latest messages the slot emitted std::vector getLatestMessagesSend() const; @@ -99,8 +95,7 @@ class Slot : public std::enable_shared_from_this SCPEnvelope const* getLatestMessage(NodeID const& id) const; // Return true if the statement is latest for a node that sent it - bool isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt); + bool isNewerStatement(SCPStatement const& oldSt, SCPStatement const& newSt); // returns messages that helped this slot externalize std::vector getExternalizingState() const; @@ -115,8 +110,6 @@ class Slot : public std::enable_shared_from_this SCP::EnvelopeState processEnvelope(SCPEnvelopeWrapperPtr envelope, bool self); - bool abandonBallot(); - // bumps the ballot based on the local state and the value passed in: // in prepare phase, attempts to take value // otherwise, no-ops @@ -124,14 +117,13 @@ class Slot : public std::enable_shared_from_this // the state if no value was prepared bool bumpState(Value const& value, bool force); - // attempts to nominate a value for consensus - bool nominate(NominationValueSupplier const& makeValue, - Value const& previousValue, bool timedout); - - void stopNomination(); + // Starts leader rounds for this slot (the local node is ready to take + // part in consensus on it). `makeValue` is invoked only if the local node + // becomes the leader of a round and has no value to put forward. + bool startLeaderRounds(ProposalValueSupplier const& makeValue); - // returns the current nomination leaders - std::set getNominationLeaders() const; + // The round the local node is in (0 if it has not started the slot). + uint32 getLeaderRound() const; bool isFullyValidated() const; void setFullyValidated(bool fullyValidated); @@ -196,14 +188,10 @@ class Slot : public std::enable_shared_from_this enum timerIDs { - NOMINATION_TIMER = 0, + // Runs while the local node waits for the leader of its current round + // to put a value forward. + LEADER_TIMER = 0, BALLOT_PROTOCOL_TIMER = 1, -#ifdef BUILD_TESTS - // Test-only: defers broadcast of a nomination vote by - // ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING. Used to simulate - // slow nominators without breaking the protocol. - NOMINATION_EMIT_TIMER = 2, -#endif }; // The number of times the timer has to expire before we consider the node diff --git a/src/scp/test/SCPTests.cpp b/src/scp/test/SCPTests.cpp index 8f57dda9b3..6feb5d7638 100644 --- a/src/scp/test/SCPTests.cpp +++ b/src/scp/test/SCPTests.cpp @@ -17,6 +17,10 @@ #include "xdrpp/marshal.h" #include "xdrpp/printer.h" #include +#include +#include +#include +#include // General convention in this file is that numbers in parenthesis // refer to the rule number in the related protocol from the white paper @@ -41,18 +45,15 @@ class TestSCP : public SCPDriver SCP mSCP; uint32_t mInitialBallotTimeoutMS = 1000; uint32_t mIncrementBallotTimeoutMS = 1000; - uint32_t mInitialNominationTimeoutMS = 1000; - uint32_t mIncrementNominationTimeoutMS = 1000; TestSCP(NodeID const& nodeID, SCPQuorumSet const& qSetLocal, bool isValidator = true) : mSCP(*this, nodeID, isValidator, qSetLocal) { - mPriorityLookup = [&](NodeID const& n) { - return (n == mSCP.getLocalNodeID()) ? 1000 : 1; - }; - - mHashValueCalculator = [&](Value const& v) { return 0; }; + // By default the local node leads every round: a ballot timeout then + // bumps the counter with the value the node already votes for, like + // the plain ballot protocol. + mRoundLeader = [&](uint64, uint32) { return mSCP.getLocalNodeID(); }; auto localQSet = std::make_shared(mSCP.getLocalQuorumSet()); @@ -73,11 +74,11 @@ class TestSCP : public SCPDriver SCPDriver::ValidationLevel validateValue(uint64 slotIndex, Value const& value, - bool nomination) const override + bool proposal) const override { if (mValidateValueOverride) { - return mValidateValueOverride(slotIndex, value, nomination); + return mValidateValueOverride(slotIndex, value, proposal); } // If we're tracking download wait time for this value, it's awaiting // download @@ -178,62 +179,55 @@ class TestSCP : public SCPDriver mEnvs.push_back(envelope); } - // used to test BallotProtocol and bypass nomination + // used to test BallotProtocol directly, without a round leader bool bumpState(uint64 slotIndex, Value const& v) { return mSCP.getSlot(slotIndex, true)->bumpState(v, true); } + // starts the slot the way the application does at its trigger; `value` + // is what the local node proposes if it leads a round bool - nominate(uint64 slotIndex, Value const& value, bool timedout) + startSlot(uint64 slotIndex, Value const& value) { auto wv = wrapValue(value); - return mSCP.getSlot(slotIndex, true) - ->nominate([wv]() { return wv; }, value, timedout); + return mSCP.startSlot(slotIndex, [this, wv]() { + ++mValuesBuilt; + return mMakeValueOverride ? mMakeValueOverride() : wv; + }); } - // only used by nomination protocol - ValueWrapperPtr - combineCandidates(uint64 slotIndex, - ValueWrapperPtrSet const& candidates) override + std::optional + getRoundLeader(uint64 slotIndex, uint32 round) override { - REQUIRE(candidates.size() == mExpectedCandidates.size()); - auto it1 = candidates.begin(); - auto it2 = mExpectedCandidates.end(); - for (; it1 != candidates.end() && it2 != mExpectedCandidates.end(); - it1++, it2++) - { - REQUIRE((*it1)->getValue() == *it2); - } - - REQUIRE(!mCompositeValue.empty()); - - return wrapValue(mCompositeValue); + return mRoundLeader(slotIndex, round); } - bool - hasUpgrades(Value const& v) override + void + proposedValue(uint64 slotIndex, uint32 round, Value const& value) override { - // Not implemented - releaseAssert(false); + mProposedValues.emplace_back(round, value); } - ValueWrapperPtr - stripAllUpgrades(Value const& v) override + void + enteredRound(uint64 slotIndex, uint32 round, bool timedOut) override { - // Not implemented - releaseAssert(false); + mEnteredRounds.emplace_back(round, timedOut); } - uint32_t - getUpgradeNominationTimeoutLimit() const override + bool + isProposalPending(uint64, NodeID const&) const override { - return std::numeric_limits::max(); + return mProposalPending; } - std::set mExpectedCandidates; - Value mCompositeValue; + std::function(uint64, uint32)> mRoundLeader; + bool mProposalPending{false}; + std::function mMakeValueOverride; + int mValuesBuilt{0}; + std::vector> mProposedValues; + std::vector> mEnteredRounds; Hash getHashOf(std::vector> const& vals) const override @@ -246,34 +240,6 @@ class TestSCP : public SCPDriver return hasher.finish(); } - // override the internal hashing scheme in order to make tests - // more predictable. - uint64 - computeHashNode(uint64 slotIndex, Value const& prev, bool isPriority, - int32_t roundNumber, NodeID const& nodeID) override - { - uint64 res; - if (isPriority) - { - res = mPriorityLookup(nodeID); - } - else - { - res = 0; - } - return res; - } - - // override the value hashing, to make tests more predictable. - uint64 - computeValueHash(uint64 slotIndex, Value const& prev, int32_t roundNumber, - Value const& value) override - { - return mHashValueCalculator(value); - } - - std::function mPriorityLookup; - std::function mHashValueCalculator; std::function mValidateValueOverride; @@ -293,15 +259,6 @@ class TestSCP : public SCPDriver std::map mTimers; std::chrono::milliseconds mCurrentTimerOffset{0}; - // Defers this node's nomination emits, like - // ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING does for Herder. - std::chrono::milliseconds mNominationEmitDelay{0}; - std::chrono::milliseconds - getNominationEmitDelayForTesting() const override - { - return mNominationEmitDelay; - } - void setupTimer(uint64 slotIndex, int timerID, std::chrono::milliseconds timeout, std::function cb) override @@ -333,6 +290,34 @@ class TestSCP : public SCPDriver return mCurrentTimerOffset; } + TimerData + getLeaderTimer() + { + return mTimers[Slot::LEADER_TIMER]; + } + + bool + hasLeaderTimer() + { + return !!getLeaderTimer().mCallback; + } + + // fires a timer the way a real (one-shot) timer does + void + fireTimer(int timerID) + { + auto cb = mTimers[timerID].mCallback; + REQUIRE(cb); + mTimers[timerID].mCallback = nullptr; + cb(); + } + + uint32 + getLeaderRound(uint64 slotIndex) + { + return mSCP.getLeaderRound(slotIndex); + } + // returns true if a ballot protocol timer exists (in the past or future) bool hasBallotTimer() @@ -351,14 +336,6 @@ class TestSCP : public SCPDriver return mCurrentTimerOffset < getBallotProtocolTimer().mAbsoluteTimeout; } - Value const& - getLatestCompositeCandidate(uint64 slotIndex) - { - return mSCP.getSlot(slotIndex, true) - ->getLatestCompositeCandidate() - ->getValue(); - } - SCP::EnvelopeState receiveEnvelope(SCPEnvelope const& envelope) { @@ -393,12 +370,6 @@ class TestSCP : public SCPDriver throw std::runtime_error("not found"); } - std::set - getNominationLeaders(uint64 slotIndex) - { - return mSCP.getSlot(slotIndex, false)->getNominationLeaders(); - } - // Helper methods for empty-tx-set value testing void startDownload(Value const& v, std::chrono::milliseconds waitTime) @@ -416,21 +387,10 @@ class TestSCP : public SCPDriver static uint32_t const MAX_TIMEOUT_MS = (30 * 60) * 1000; std::chrono::milliseconds - computeTimeout(uint32 roundNumber, bool isNomination) override + computeTimeout(uint32 roundNumber) override { - int initialTimeoutMS; - int incrementMS; - - if (isNomination) - { - initialTimeoutMS = mInitialNominationTimeoutMS; - incrementMS = mIncrementNominationTimeoutMS; - } - else - { - initialTimeoutMS = mInitialBallotTimeoutMS; - incrementMS = mIncrementBallotTimeoutMS; - } + int initialTimeoutMS = mInitialBallotTimeoutMS; + int incrementMS = mIncrementBallotTimeoutMS; int timeoutMS = initialTimeoutMS + (roundNumber - 1) * incrementMS; if (timeoutMS > MAX_TIMEOUT_MS) @@ -599,15 +559,6 @@ verifyExternalize(SCPEnvelope const& actual, SecretKey const& secretKey, REQUIRE(exp.statement == actual.statement); } -void -verifyNominate(SCPEnvelope const& actual, SecretKey const& secretKey, - Hash const& qSetHash, uint64 slotIndex, std::vector votes, - std::vector accepted) -{ - auto exp = makeNominate(secretKey, qSetHash, slotIndex, votes, accepted); - REQUIRE(exp.statement == actual.statement); -} - // Simulate xValue being only structurally valid (e.g., due to an invalid tx // set) SCPDriver::ValidationLevel @@ -803,12 +754,8 @@ testTimeouts(TestSCP& scp, std::function f) { SECTION("minimum values") { - scp.mInitialNominationTimeoutMS = MinimumSorobanNetworkConfig:: - NOMINATION_TIMEOUT_INITIAL_MILLISECONDS; scp.mInitialBallotTimeoutMS = MinimumSorobanNetworkConfig::BALLOT_TIMEOUT_INITIAL_MILLISECONDS; - scp.mIncrementNominationTimeoutMS = MinimumSorobanNetworkConfig:: - NOMINATION_TIMEOUT_INCREMENT_MILLISECONDS; scp.mIncrementBallotTimeoutMS = MinimumSorobanNetworkConfig::BALLOT_TIMEOUT_INCREMENT_MILLISECONDS; f(scp); @@ -816,12 +763,8 @@ testTimeouts(TestSCP& scp, std::function f) SECTION("initial values") { - scp.mInitialNominationTimeoutMS = InitialSorobanNetworkConfig:: - NOMINATION_TIMEOUT_INITIAL_MILLISECONDS; scp.mInitialBallotTimeoutMS = InitialSorobanNetworkConfig::BALLOT_TIMEOUT_INITIAL_MILLISECONDS; - scp.mIncrementNominationTimeoutMS = InitialSorobanNetworkConfig:: - NOMINATION_TIMEOUT_INCREMENT_MILLISECONDS; scp.mIncrementBallotTimeoutMS = InitialSorobanNetworkConfig::BALLOT_TIMEOUT_INCREMENT_MILLISECONDS; f(scp); @@ -829,12 +772,8 @@ testTimeouts(TestSCP& scp, std::function f) SECTION("maximum values") { - scp.mInitialNominationTimeoutMS = MaximumSorobanNetworkConfig:: - NOMINATION_TIMEOUT_INITIAL_MILLISECONDS; scp.mInitialBallotTimeoutMS = MaximumSorobanNetworkConfig::BALLOT_TIMEOUT_INITIAL_MILLISECONDS; - scp.mIncrementNominationTimeoutMS = MaximumSorobanNetworkConfig:: - NOMINATION_TIMEOUT_INCREMENT_MILLISECONDS; scp.mIncrementBallotTimeoutMS = MaximumSorobanNetworkConfig::BALLOT_TIMEOUT_INCREMENT_MILLISECONDS; f(scp); @@ -2900,663 +2839,218 @@ TEST_CASE("ballot protocol core3", "[scp][ballotprotocol]") testTimeouts(scp, test); } -TEST_CASE("delayed nomination emit after an immediate emit", - "[scp][nominationprotocol]") +TEST_CASE("ballot protocol self-emits CONFIRM after federated accept-commit on " + "structurally-valid value", + "[scp][ballotprotocol]") { setupValues(); SIMULATION_CREATE_NODE(0); SIMULATION_CREATE_NODE(1); SIMULATION_CREATE_NODE(2); - SIMULATION_CREATE_NODE(3); - SIMULATION_CREATE_NODE(4); - // v-blocking set size 2, quorum threshold 4 SCPQuorumSet qSet; - qSet.threshold = 4; + qSet.threshold = 2; qSet.validators.push_back(v0NodeID); qSet.validators.push_back(v1NodeID); qSet.validators.push_back(v2NodeID); - qSet.validators.push_back(v3NodeID); - qSet.validators.push_back(v4NodeID); + uint256 qSetHash = sha256(xdr::xdr_to_opaque(qSet)); TestSCP scp(v0SecretKey.getPublicKey(), qSet); scp.storeQuorumSet(std::make_shared(qSet)); - uint256 qSetHash0 = scp.mSCP.getLocalNode()->getQuorumSetHash(); - scp.mNominationEmitDelay = std::chrono::milliseconds(1000); - // v0 leads the round and votes for x, but its emit is deferred - REQUIRE(scp.nominate(0, xValue, false)); - REQUIRE(scp.mEnvs.empty()); - REQUIRE(scp.mTimers.count(Slot::NOMINATION_EMIT_TIMER) == 1); - auto delayedEmit = scp.mTimers[Slot::NOMINATION_EMIT_TIMER].mCallback; - REQUIRE(delayedEmit); + // xValue stays kStructurallyValidValue throughout, with a wait time + // below the download timeout so maybeReplaceValueWithEmptyTxSet does not + // replace it with an empty-tx-set value. + scp.startDownload(xValue, UNDER_TX_SET_TIMEOUT); + scp.mValidateValueOverride = xValueStructurallyValidValidationOverride; - std::vector votes{xValue}; + SCPBallot xB1(1, xValue); - SECTION("nothing emitted meanwhile: the delayed emit sends the vote") - { - delayedEmit(); - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, votes, {}); - } + // v0 enters ballot protocol with xValue. + REQUIRE(scp.bumpState(0, xValue)); + REQUIRE(scp.mEnvs.size() == 1); - SECTION("an immediate emit meanwhile: the delayed emit is not repeated") - { - // A v-blocking set accepting x makes v0 accept it, and processEnvelope - // emits the new state at once while the delayed emit is still armed. - std::vector accepted{xValue}; - scp.receiveEnvelope( - makeNominate(v1SecretKey, qSetHash, 0, votes, accepted)); - REQUIRE(scp.mEnvs.empty()); - scp.receiveEnvelope( - makeNominate(v2SecretKey, qSetHash, 0, votes, accepted)); - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, votes, - accepted); + // v1, v2 vote-prepare for (1, xValue) → v0 accept-prepared (1, xValue). + REQUIRE(scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1)) == + SCP::EnvelopeState::VALID); + REQUIRE(scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, xB1)) == + SCP::EnvelopeState::VALID); - // Nothing changed since: re-emitting the same statement used to be - // rejected by our own processEnvelope ("moved to a bad state"). - REQUIRE_NOTHROW(delayedEmit()); - REQUIRE(scp.mEnvs.size() == 1); - } + // v1, v2 signal accept-prepared (1, xValue). v0 would normally + // confirm-prepared here, but setConfirmPrepared stalls on + // kStructurallyValidValue so c/h stay unset on v0's side. + REQUIRE( + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1, &xB1)) == + SCP::EnvelopeState::VALID); + REQUIRE( + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, xB1, &xB1)) == + SCP::EnvelopeState::VALID); + + // v1, v2 vote-to-commit (1, xValue) via nC/nH on their PREPAREs. + // federatedAccept fires via the "quorum voted-or-accepted" path v0 + // accept-commits, transitions mPhase to CONFIRM, and self-emits a CONFIRM + // with xValue. + REQUIRE(scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, xB1, &xB1, 1, 1)) == + SCP::EnvelopeState::VALID); + REQUIRE(scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, xB1, &xB1, 1, 1)) == + SCP::EnvelopeState::VALID); + + // Last emitted envelope should be a CONFIRM carrying xValue — proves that + // processEnvelope correctly accepts self-emitted CONFIRMs with + // kStructurallyValidValue. + auto const& lastEnv = scp.mEnvs.back(); + REQUIRE(lastEnv.statement.pledges.type() == SCP_ST_CONFIRM); + auto const& cBallot = lastEnv.statement.pledges.confirm().ballot; + REQUIRE(cBallot.value == xValue); + REQUIRE(!scp.isEmptyTxSetValue(cBallot.value)); + + // A peer CONFIRM whose value v0 considers kStructurallyValidValue is + // rejected by processEnvelope + auto const res = scp.receiveEnvelope( + makeConfirm(v1SecretKey, qSetHash, 0, 1, xB1, 1, 1)); + REQUIRE(res == SCP::EnvelopeState::INVALID); } -TEST_CASE("nomination tests core5", "[scp][nominationprotocol]") +TEST_CASE("drop tx set on download timeout", "[scp][ballotprotocol]") { setupValues(); SIMULATION_CREATE_NODE(0); SIMULATION_CREATE_NODE(1); SIMULATION_CREATE_NODE(2); - SIMULATION_CREATE_NODE(3); - SIMULATION_CREATE_NODE(4); - // we need 5 nodes to avoid sharing various thresholds: - // v-blocking set size: 2 - // threshold: 4 = 3 + self or 4 others + // 3 node network with threshold=2 (need any 2 nodes to form quorum) SCPQuorumSet qSet; - qSet.threshold = 4; + qSet.threshold = 2; qSet.validators.push_back(v0NodeID); qSet.validators.push_back(v1NodeID); qSet.validators.push_back(v2NodeID); - qSet.validators.push_back(v3NodeID); - qSet.validators.push_back(v4NodeID); uint256 qSetHash = sha256(xdr::xdr_to_opaque(qSet)); - REQUIRE(xValue < yValue); - REQUIRE(yValue < zValue); - - auto checkLeaders = [&](TestSCP& scp, std::set expectedLeaders) { - auto l = scp.getNominationLeaders(0); - REQUIRE(std::equal(l.begin(), l.end(), expectedLeaders.begin(), - expectedLeaders.end())); - }; + TestSCP scp(v0SecretKey.getPublicKey(), qSet); + scp.storeQuorumSet(std::make_shared(qSet)); + uint256 qSetHash0 = scp.mSCP.getLocalNode()->getQuorumSetHash(); - SECTION("nomination - v0 is top") + SECTION("timeout during prepare phase") { - TestSCP scp(v0SecretKey.getPublicKey(), qSet); + // Node v0 starts ballot protocol with xValue + // Simulate that xValue is awaiting download with timeout exceeded + scp.startDownload(xValue, OVER_TX_SET_TIMEOUT); - auto test = [&](TestSCP& scp) { - uint256 qSetHash0 = scp.mSCP.getLocalNode()->getQuorumSetHash(); - scp.storeQuorumSet(std::make_shared(qSet)); + // Now call bumpState which should trigger + // maybeReplaceValueWithEmptyTxSet + REQUIRE(scp.bumpState(0, xValue)); + REQUIRE(scp.mEnvs.size() == 1); - SECTION("v0 starts to nominates xValue") - { - REQUIRE(scp.nominate(0, xValue, false)); + // The ballot should have an empty-tx-set value, not the original + // xValue + auto const& emittedBallot = + scp.mEnvs[0].statement.pledges.prepare().ballot; - checkLeaders(scp, {v0SecretKey.getPublicKey()}); + REQUIRE(emittedBallot.counter == 1); + REQUIRE(scp.isEmptyTxSetValue(emittedBallot.value)); - SECTION("others nominate what v0 says (x) -> prepare x") - { - std::vector votes, accepted; - votes.emplace_back(xValue); + // Verify it's the empty-tx-set value derived from xValue + Value expectedEmptyTxSetValue = + scp.makeEmptyTxSetValueFromValue(xValue); + REQUIRE(emittedBallot.value == expectedEmptyTxSetValue); - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, - votes, accepted); - - SCPEnvelope nom1 = - makeNominate(v1SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope nom2 = - makeNominate(v2SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope nom3 = - makeNominate(v3SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope nom4 = - makeNominate(v4SecretKey, qSetHash, 0, votes, accepted); - - // nothing happens yet - scp.receiveEnvelope(nom1); - scp.receiveEnvelope(nom2); - REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, + SCPBallot(1, expectedEmptyTxSetValue)); + } - // this causes 'x' to be accepted (quorum) - scp.receiveEnvelope(nom3); - REQUIRE(scp.mEnvs.size() == 2); + SECTION("no timeout when wait time under threshold") + { + // Node v0 starts ballot protocol with xValue + REQUIRE(scp.bumpState(0, xValue)); + REQUIRE(scp.mEnvs.size() == 1); - scp.mExpectedCandidates.emplace(xValue); - scp.mCompositeValue = xValue; + SCPBallot b1(1, xValue); - accepted.emplace_back(xValue); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, - votes, accepted); + // Simulate that xValue is awaiting download but wait time is still low + scp.startDownload(xValue, UNDER_TX_SET_TIMEOUT); - // extra message doesn't do anything - scp.receiveEnvelope(nom4); - REQUIRE(scp.mEnvs.size() == 2); + // Try to bump state - should NOT replace with an empty-tx-set value + REQUIRE(scp.bumpState(0, xValue)); + REQUIRE(scp.mEnvs.size() == 2); - SCPEnvelope acc1 = - makeNominate(v1SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope acc2 = - makeNominate(v2SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope acc3 = - makeNominate(v3SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope acc4 = - makeNominate(v4SecretKey, qSetHash, 0, votes, accepted); - - // nothing happens yet - scp.receiveEnvelope(acc1); - scp.receiveEnvelope(acc2); - REQUIRE(scp.mEnvs.size() == 2); + // Verify ballot still has original xValue, not an empty-tx-set value + auto const& emittedBallot = + scp.mEnvs[1].statement.pledges.prepare().ballot; + REQUIRE(emittedBallot.counter == 2); + REQUIRE(!scp.isEmptyTxSetValue(emittedBallot.value)); + REQUIRE(emittedBallot.value == xValue); + } - scp.mCompositeValue = xValue; - // this causes the node to send a prepare message (quorum) - scp.receiveEnvelope(acc3); - REQUIRE(scp.mEnvs.size() == 3); + SECTION("empty-tx-set value can be prepared and confirmed") + { + // Start with xValue and timeout to an empty-tx-set value + scp.startDownload(xValue, OVER_TX_SET_TIMEOUT); - verifyPrepare(scp.mEnvs[2], v0SecretKey, qSetHash0, 0, - SCPBallot(1, xValue)); + REQUIRE(scp.bumpState(0, xValue)); + REQUIRE(scp.mEnvs.size() == 1); - scp.receiveEnvelope(acc4); - REQUIRE(scp.mEnvs.size() == 3); + Value emptyTxSetValue = scp.makeEmptyTxSetValueFromValue(xValue); + SCPBallot emptyTxSetB1(1, emptyTxSetValue); - std::vector votes2 = votes; - votes2.emplace_back(yValue); + // Verify we emitted an empty-tx-set value + REQUIRE(scp.isEmptyTxSetValue( + scp.mEnvs[0].statement.pledges.prepare().ballot.value)); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, emptyTxSetB1); - SECTION("nominate x -> accept x -> prepare (x) ; others " - "accepted y " - "-> update latest to (z=x+y)") - { - SCPEnvelope acc1_2 = makeNominate(v1SecretKey, qSetHash, - 0, votes2, votes2); - SCPEnvelope acc2_2 = makeNominate(v2SecretKey, qSetHash, - 0, votes2, votes2); - SCPEnvelope acc3_2 = makeNominate(v3SecretKey, qSetHash, - 0, votes2, votes2); - SCPEnvelope acc4_2 = makeNominate(v4SecretKey, qSetHash, - 0, votes2, votes2); - - scp.receiveEnvelope(acc1_2); - REQUIRE(scp.mEnvs.size() == 3); - - // v-blocking - scp.receiveEnvelope(acc2_2); - REQUIRE(scp.mEnvs.size() == 4); - verifyNominate(scp.mEnvs[3], v0SecretKey, qSetHash0, 0, - votes2, votes2); - - scp.mExpectedCandidates.insert(yValue); - scp.mCompositeValue = kValue; - // this updates the composite value to use next time - // but does not prepare it - scp.receiveEnvelope(acc3_2); - REQUIRE(scp.mEnvs.size() == 4); - - REQUIRE(scp.getLatestCompositeCandidate(0) == kValue); - - scp.receiveEnvelope(acc4_2); - REQUIRE(scp.mEnvs.size() == 4); - } - SECTION("nomination - restored state") - { - TestSCP scp2(v0SecretKey.getPublicKey(), qSet); - scp2.storeQuorumSet( - std::make_shared(qSet)); - - // at this point - // votes = { x } - // accepted = { x } - - // tests if nomination proceeds like normal - // nominates x - auto nominationRestore = [&]() { - // restores from the previous state - scp2.mSCP.setStateFromEnvelope( - 0, scp2.wrapEnvelope( - makeNominate(v0SecretKey, qSetHash0, 0, - votes, accepted))); - // tries to start nomination with yValue, but picks - // xValue since it was already in the votes - REQUIRE(!scp2.nominate(0, yValue, false)); - - checkLeaders(scp2, {v0SecretKey.getPublicKey()}); - - REQUIRE(scp2.mEnvs.size() == 0); - - // other nodes vote for 'x' - scp2.receiveEnvelope(nom1); - scp2.receiveEnvelope(nom2); - REQUIRE(scp2.mEnvs.size() == 0); - // 'x' is accepted (quorum) - // but because the restored state already included - // 'x' in the accepted set, no new message is - // emitted - scp2.receiveEnvelope(nom3); - - scp2.mExpectedCandidates.emplace(xValue); - scp2.mCompositeValue = xValue; - - // other nodes not emit 'x' as accepted - scp2.receiveEnvelope(acc1); - scp2.receiveEnvelope(acc2); - REQUIRE(scp2.mEnvs.size() == 0); - - scp2.mCompositeValue = xValue; - // this causes the node to update its composite - // value to - // x - scp2.receiveEnvelope(acc3); - }; - - SECTION("ballot protocol not started") - { - nominationRestore(); - // nomination ended up starting the ballot protocol - REQUIRE(scp2.mEnvs.size() == 1); + // Other nodes also move to the empty-tx-set value + scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, emptyTxSetB1)); + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, emptyTxSetB1)); - verifyPrepare(scp2.mEnvs[0], v0SecretKey, qSetHash0, - 0, SCPBallot(1, xValue)); - } - SECTION("ballot protocol started (on value k)") - { - scp2.mSCP.setStateFromEnvelope( - 0, scp2.wrapEnvelope( - makePrepare(v0SecretKey, qSetHash0, 0, - SCPBallot(1, kValue)))); - nominationRestore(); - // nomination didn't do anything (already working on - // k) - REQUIRE(scp2.mEnvs.size() == 0); - } - } - } - SECTION("receive more messages, then v0 switches to a " - "different leader") - { - SCPEnvelope nom1 = - makeNominate(v1SecretKey, qSetHash, 0, {kValue}, {}); - SCPEnvelope nom2 = - makeNominate(v2SecretKey, qSetHash, 0, {yValue}, {}); - - // nothing more happens - scp.receiveEnvelope(nom1); - scp.receiveEnvelope(nom2); - REQUIRE(scp.mEnvs.size() == 1); + // Should prepare the empty-tx-set value (quorum reached) + REQUIRE(scp.mEnvs.size() == 2); + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, emptyTxSetB1, + &emptyTxSetB1); - // switch leader to v1 - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v1NodeID) ? 1000 : 1; - }; - REQUIRE(scp.nominate(0, xValue, true)); - REQUIRE(scp.mEnvs.size() == 2); + // Quorum confirms prepared empty-tx-set value + scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, emptyTxSetB1, &emptyTxSetB1)); + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, emptyTxSetB1, &emptyTxSetB1)); - std::vector votesXK; - votesXK.emplace_back(xValue); - votesXK.emplace_back(kValue); - std::sort(votesXK.begin(), votesXK.end()); + REQUIRE(scp.mEnvs.size() == 3); + verifyPrepare(scp.mEnvs[2], v0SecretKey, qSetHash0, 0, emptyTxSetB1, + &emptyTxSetB1, 1, 1); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, - votesXK, {}); - } - SECTION("select accepted value from leader") - { - REQUIRE(xValue < yValue); - REQUIRE(yValue < zValue); - REQUIRE(scp.mEnvs.size() == 1); + // Accept commit + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, emptyTxSetB1, + &emptyTxSetB1, 1, 1)); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, emptyTxSetB1, + &emptyTxSetB1, 1, 1)); - // Update round leader to v1 - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v1NodeID) ? 1000 : 1; - }; + REQUIRE(scp.mEnvs.size() == 4); + verifyConfirm(scp.mEnvs[3], v0SecretKey, qSetHash0, 0, 1, emptyTxSetB1, + 1, 1); - SCPEnvelope nom1 = makeNominate(v1SecretKey, qSetHash, 0, - {yValue, zValue}, {yValue}); + // Externalize the empty-tx-set value + scp.receiveEnvelope( + makeConfirm(v1SecretKey, qSetHash, 0, 1, emptyTxSetB1, 1, 1)); + scp.receiveEnvelope( + makeConfirm(v2SecretKey, qSetHash, 0, 1, emptyTxSetB1, 1, 1)); - SECTION("receive accepted before timeout") - { - // nothing more happens, v0 is leader - scp.receiveEnvelope(nom1); - REQUIRE(scp.mEnvs.size() == 1); - - // Update round leaders, vote for accepted value (y) - REQUIRE(scp.nominate(0, xValue, true)); - REQUIRE(scp.mEnvs.size() == 2); - } - SECTION("receive accepted after timeout") - { - REQUIRE(!scp.nominate(0, xValue, true)); - REQUIRE(scp.mEnvs.size() == 1); - - // Vote for accepted value (y) - scp.receiveEnvelope(nom1); - REQUIRE(scp.mEnvs.size() == 2); - } - - std::vector votesXY; - votesXY.emplace_back(xValue); - votesXY.emplace_back(yValue); - - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, - votesXY, {}); - - SCPEnvelope nom2 = - makeNominate(v1SecretKey, qSetHash, 0, - {yValue, zValue, zzValue}, {yValue}); - scp.receiveEnvelope(nom2); - // Nothing happens, as v0 already voted for the accepted - // value (y) - REQUIRE(scp.mEnvs.size() == 2); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, - votesXY, {}); - } - } - SECTION("self nominates 'x', others nominate y -> prepare y") - { - std::vector myVotes, accepted; - myVotes.emplace_back(xValue); - - scp.mExpectedCandidates.emplace(xValue); - scp.mCompositeValue = xValue; - REQUIRE(scp.nominate(0, xValue, false)); - - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, myVotes, - accepted); - - std::vector votes; - votes.emplace_back(yValue); - - std::vector acceptedY = accepted; - - acceptedY.emplace_back(yValue); - - SECTION("others only vote for y") - { - SCPEnvelope nom1 = - makeNominate(v1SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope nom2 = - makeNominate(v2SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope nom3 = - makeNominate(v3SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope nom4 = - makeNominate(v4SecretKey, qSetHash, 0, votes, accepted); - - // nothing happens yet - scp.receiveEnvelope(nom1); - scp.receiveEnvelope(nom2); - scp.receiveEnvelope(nom3); - REQUIRE(scp.mEnvs.size() == 1); - - // 'y' is accepted (quorum) - scp.receiveEnvelope(nom4); - REQUIRE(scp.mEnvs.size() == 2); - myVotes.emplace_back(yValue); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, - myVotes, acceptedY); - } - SECTION("others accepted y") - { - SCPEnvelope acc1 = makeNominate(v1SecretKey, qSetHash, 0, - votes, acceptedY); - SCPEnvelope acc2 = makeNominate(v2SecretKey, qSetHash, 0, - votes, acceptedY); - SCPEnvelope acc3 = makeNominate(v3SecretKey, qSetHash, 0, - votes, acceptedY); - SCPEnvelope acc4 = makeNominate(v4SecretKey, qSetHash, 0, - votes, acceptedY); - - scp.receiveEnvelope(acc1); - REQUIRE(scp.mEnvs.size() == 1); - - // this causes 'y' to be accepted (v-blocking) - scp.receiveEnvelope(acc2); - REQUIRE(scp.mEnvs.size() == 2); - - myVotes.emplace_back(yValue); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, - myVotes, acceptedY); - - scp.mExpectedCandidates.clear(); - scp.mExpectedCandidates.insert(yValue); - scp.mCompositeValue = yValue; - // this causes the node to send a prepare message (quorum) - scp.receiveEnvelope(acc3); - REQUIRE(scp.mEnvs.size() == 3); - - verifyPrepare(scp.mEnvs[2], v0SecretKey, qSetHash0, 0, - SCPBallot(1, yValue)); - - scp.receiveEnvelope(acc4); - REQUIRE(scp.mEnvs.size() == 3); - } - } - }; - - testTimeouts(scp, test); - } - SECTION("v1 is top node") - { - TestSCP scp(v0SecretKey.getPublicKey(), qSet); - - auto test = [&](TestSCP& scp) { - uint256 qSetHash0 = scp.mSCP.getLocalNode()->getQuorumSetHash(); - scp.storeQuorumSet(std::make_shared(qSet)); - - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v1NodeID) ? 1000 : 1; - }; - - std::vector votesX, votesY, votesK, votesXY, votesYK, - votesXK, emptyV; - votesX.emplace_back(xValue); - votesY.emplace_back(yValue); - votesK.emplace_back(kValue); - - votesXY.emplace_back(xValue); - votesXY.emplace_back(yValue); - - votesYK.emplace_back(yValue); - votesYK.emplace_back(kValue); - std::sort(votesYK.begin(), votesYK.end()); - - votesXK.emplace_back(xValue); - votesXK.emplace_back(kValue); - std::sort(votesXK.begin(), votesXK.end()); - - std::vector valuesHash; - valuesHash.emplace_back(xValue); - valuesHash.emplace_back(yValue); - valuesHash.emplace_back(kValue); - std::sort(valuesHash.begin(), valuesHash.end()); - - scp.mHashValueCalculator = [&](Value const& v) { - auto pos = std::find(valuesHash.begin(), valuesHash.end(), v); - if (pos == valuesHash.end()) - { - abort(); - } - return 1 + std::distance(valuesHash.begin(), pos); - }; - - SCPEnvelope nom1 = - makeNominate(v1SecretKey, qSetHash, 0, votesXY, emptyV); - SCPEnvelope nom2 = - makeNominate(v2SecretKey, qSetHash, 0, votesXK, emptyV); - - SECTION( - "value from v1 is a candidate, self should not introduce new " - "value on timeout") - { - REQUIRE(!scp.nominate(0, xValue, false)); - checkLeaders(scp, {v1SecretKey.getPublicKey()}); - - REQUIRE(scp.mEnvs.size() == 0); - nom1 = makeNominate(v1SecretKey, qSetHash, 0, votesX, emptyV); - nom2 = makeNominate(v2SecretKey, qSetHash, 0, votesX, emptyV); - SCPEnvelope nom3 = - makeNominate(v3SecretKey, qSetHash, 0, votesX, emptyV); - - // Receive `x` from v1, vote for it - scp.receiveEnvelope(nom1); - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, votesX, - emptyV); - - scp.receiveEnvelope(nom2); - scp.receiveEnvelope(nom3); - REQUIRE(scp.mEnvs.size() == 2); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, votesX, - votesX); - - SCPEnvelope acc1 = - makeNominate(v1SecretKey, qSetHash, 0, votesX, votesX); - SCPEnvelope acc2 = - makeNominate(v2SecretKey, qSetHash, 0, votesX, votesX); - SCPEnvelope acc3 = - makeNominate(v3SecretKey, qSetHash, 0, votesX, votesX); - - scp.receiveEnvelope(acc1); - scp.receiveEnvelope(acc2); - REQUIRE(scp.mEnvs.size() == 2); - - // Receive accept from quorum, ratify and generate a candidate - // value - REQUIRE(scp.mTimers.find(Slot::NOMINATION_TIMER) != - scp.mTimers.end()); - scp.mCompositeValue = xValue; - scp.mExpectedCandidates.emplace(xValue); - scp.receiveEnvelope(acc3); - REQUIRE(scp.mEnvs.size() == 3); - // Timer is cancelled - REQUIRE(scp.mTimers.find(Slot::NOMINATION_TIMER) == - scp.mTimers.end()); - - // v0 is the new leader, but we already have a candidate - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v0NodeID) ? 1000 : 1; - }; - REQUIRE(!scp.nominate(0, kValue, true)); - } - SECTION("nomination waits for v1") - { - REQUIRE(!scp.nominate(0, xValue, false)); - - checkLeaders(scp, {v1SecretKey.getPublicKey()}); - - REQUIRE(scp.mEnvs.size() == 0); - - SCPEnvelope nom4 = - makeNominate(v4SecretKey, qSetHash, 0, votesXK, emptyV); - - // nothing happens with non top nodes - scp.receiveEnvelope(nom2); - // (note: don't receive anything from node3 - we want to pick - // another dead node) - REQUIRE(scp.mEnvs.size() == 0); - - // v1 is leader -> nominate the first value from its message - // that's "y" - scp.receiveEnvelope(nom1); - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, votesY, - emptyV); - - scp.receiveEnvelope(nom4); - REQUIRE(scp.mEnvs.size() == 1); - - // "timeout -> pick another value from v1" - scp.mExpectedCandidates.emplace(xValue); - scp.mCompositeValue = xValue; - - // allows to pick another leader, - // pick another dead node v3 as to force picking up - // a new value from v1 - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v3NodeID) ? 1000 : 1; - }; - - // note: value passed in here should be ignored - REQUIRE(scp.nominate(0, kValue, true)); - // picks up 'x' from v1 (as we already have 'y') - // which also happens to causes 'x' to be accepted - REQUIRE(scp.mEnvs.size() == 2); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, votesXY, - votesX); - } - SECTION("v1 dead, timeout") - { - REQUIRE(!scp.nominate(0, xValue, false)); - - REQUIRE(scp.mEnvs.size() == 0); - - scp.receiveEnvelope(nom2); - REQUIRE(scp.mEnvs.size() == 0); - - checkLeaders(scp, {v1SecretKey.getPublicKey()}); - - SECTION("v0 is new top node") - { - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v0NodeID) ? 1000 : 1; - }; - - REQUIRE(scp.nominate(0, xValue, true)); - checkLeaders(scp, {v0SecretKey.getPublicKey(), - v1SecretKey.getPublicKey()}); - - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, - votesX, emptyV); - } - SECTION("v2 is new top node") - { - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v2NodeID) ? 1000 : 1; - }; - - REQUIRE(scp.nominate(0, xValue, true)); - checkLeaders(scp, {v1SecretKey.getPublicKey(), - v2SecretKey.getPublicKey()}); - - REQUIRE(scp.mEnvs.size() == 1); - // v2 votes for XK, but nomination only picks the highest - // value - std::vector v2Top; - v2Top.emplace_back(std::max(xValue, kValue)); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, - v2Top, emptyV); - } - SECTION("v3 is new top node") - { - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v3NodeID) ? 1000 : 1; - }; - // nothing happens, we don't have any message for v3 - REQUIRE(!scp.nominate(0, xValue, true)); - checkLeaders(scp, {v1SecretKey.getPublicKey(), - v3SecretKey.getPublicKey()}); - - REQUIRE(scp.mEnvs.size() == 0); - } - } - }; + REQUIRE(scp.mEnvs.size() == 5); + verifyExternalize(scp.mEnvs[4], v0SecretKey, qSetHash0, 0, emptyTxSetB1, + 1); - testTimeouts(scp, test); + // Verify the externalized value is the empty-tx-set value + REQUIRE(scp.mExternalizedValues.size() == 1); + REQUIRE(scp.isEmptyTxSetValue(scp.mExternalizedValues[0])); + REQUIRE(scp.mExternalizedValues[0] == emptyTxSetValue); } } -TEST_CASE("nomination times out structurally-valid value into empty tx set", - "[scp][nomination]") +TEST_CASE("Proper handling of non-current ledger value", + "[scp][ballotprotocol]") { setupValues(); SIMULATION_CREATE_NODE(0); @@ -3569,48 +3063,19 @@ TEST_CASE("nomination times out structurally-valid value into empty tx set", qSet.validators.push_back(v1NodeID); qSet.validators.push_back(v2NodeID); - auto const qSetHash = sha256(xdr::xdr_to_opaque(qSet)); - TestSCP scp(v0SecretKey.getPublicKey(), qSet); scp.storeQuorumSet(std::make_shared(qSet)); - // xValue is structurally-valid throughout — its tx set is either still in - // flight or has been downloaded-and-found-invalid. Seed a wait time past - // the download timeout so maybeReplaceValueWithEmptyTxSet triggers - // empty-tx-set replacement at bumpState time. - scp.startDownload(xValue, OVER_TX_SET_TIMEOUT); - scp.mValidateValueOverride = xValueStructurallyValidValidationOverride; - - REQUIRE(scp.nominate(0, xValue, false)); - - auto const followerVoteNomination = - makeNominate(v1SecretKey, qSetHash, 0, {xValue}, {}); - REQUIRE(scp.receiveEnvelope(followerVoteNomination) == - SCP::EnvelopeState::VALID); - - scp.mExpectedCandidates.emplace(xValue); - scp.mCompositeValue = xValue; + // validateValue reports xValue is not for the current ledger. + scp.mValidateValueOverride = xValueNotCurrentLedgerOverride; - auto const followerAcceptedNomination = - makeNominate(v2SecretKey, qSetHash, 0, {xValue}, {xValue}); - // Quorum accept-nominated xValue → composite value flows to - // bumpState → maybeReplaceValueWithEmptyTxSet sees - // kStructurallyValidValue with the wait time past the timeout and - // substitutes an empty-tx-set value. - REQUIRE(scp.receiveEnvelope(followerAcceptedNomination) == - SCP::EnvelopeState::VALID); + REQUIRE(scp.bumpState(0, xValue)); - // The emitted ballot should carry the empty-tx-set value derived from - // xValue, not xValue itself. - auto const& lastEnv = scp.mEnvs.back(); - REQUIRE(lastEnv.statement.pledges.type() == SCP_ST_PREPARE); - auto const& ballot = lastEnv.statement.pledges.prepare().ballot; - REQUIRE(scp.isEmptyTxSetValue(ballot.value)); - REQUIRE(ballot.value == scp.makeEmptyTxSetValueFromValue(xValue)); + // Do not emit a ballot with a kMaybeValid value + REQUIRE(scp.mEnvs.size() == 0); } -TEST_CASE("ballot protocol self-emits CONFIRM after federated accept-commit on " - "structurally-valid value", +TEST_CASE("setConfirmPrepared stalls on kStructurallyValidValue value", "[scp][ballotprotocol]") { setupValues(); @@ -3629,27 +3094,15 @@ TEST_CASE("ballot protocol self-emits CONFIRM after federated accept-commit on " TestSCP scp(v0SecretKey.getPublicKey(), qSet); scp.storeQuorumSet(std::make_shared(qSet)); - // xValue stays kStructurallyValidValue throughout, with a wait time - // below the download timeout so maybeReplaceValueWithEmptyTxSet does not - // replace it with an empty-tx-set value. + // Simulate parallel downloading scp.startDownload(xValue, UNDER_TX_SET_TIMEOUT); - scp.mValidateValueOverride = xValueStructurallyValidValidationOverride; - - SCPBallot xB1(1, xValue); - // v0 enters ballot protocol with xValue. + // v0 enters ballot protocol REQUIRE(scp.bumpState(0, xValue)); REQUIRE(scp.mEnvs.size() == 1); + SCPBallot xB1(1, xValue); - // v1, v2 vote-prepare for (1, xValue) → v0 accept-prepared (1, xValue). - REQUIRE(scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1)) == - SCP::EnvelopeState::VALID); - REQUIRE(scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, xB1)) == - SCP::EnvelopeState::VALID); - - // v1, v2 signal accept-prepared (1, xValue). v0 would normally - // confirm-prepared here, but setConfirmPrepared stalls on - // kStructurallyValidValue so c/h stay unset on v0's side. + // v1 and v2 send PREPAREs with prepared — quorum confirms prepared REQUIRE( scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1, &xB1)) == SCP::EnvelopeState::VALID); @@ -3657,41 +3110,72 @@ TEST_CASE("ballot protocol self-emits CONFIRM after federated accept-commit on " scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, xB1, &xB1)) == SCP::EnvelopeState::VALID); - // v1, v2 vote-to-commit (1, xValue) via nC/nH on their PREPAREs. - // federatedAccept fires via the "quorum voted-or-accepted" path v0 - // accept-commits, transitions mPhase to CONFIRM, and self-emits a CONFIRM - // with xValue. - REQUIRE(scp.receiveEnvelope( - makePrepare(v1SecretKey, qSetHash, 0, xB1, &xB1, 1, 1)) == - SCP::EnvelopeState::VALID); - REQUIRE(scp.receiveEnvelope( - makePrepare(v2SecretKey, qSetHash, 0, xB1, &xB1, 1, 1)) == - SCP::EnvelopeState::VALID); + SECTION("commit gate stalls mCommit but mHighBallot is set") + { - // Last emitted envelope should be a CONFIRM carrying xValue — proves that - // processEnvelope correctly accepts self-emitted CONFIRMs with - // kStructurallyValidValue. - auto const& lastEnv = scp.mEnvs.back(); - REQUIRE(lastEnv.statement.pledges.type() == SCP_ST_CONFIRM); - auto const& cBallot = lastEnv.statement.pledges.confirm().ballot; - REQUIRE(cBallot.value == xValue); - REQUIRE(!scp.isEmptyTxSetValue(cBallot.value)); + // setConfirmPrepared sets mHighBallot (nH > 0) but the commit gate + // stalls mCommit (nC == 0) because xValue is kStructurallyValidValue. + // Check the latest emitted PREPARE. + REQUIRE(scp.mEnvs.size() >= 2); + auto const& lastPrep = scp.mEnvs.back().statement.pledges.prepare(); + REQUIRE(lastPrep.nH == 1); + REQUIRE(lastPrep.nC == 0); + } - // A peer CONFIRM whose value v0 considers kStructurallyValidValue is - // rejected by processEnvelope - auto const res = scp.receiveEnvelope( - makeConfirm(v1SecretKey, qSetHash, 0, 1, xB1, 1, 1)); - REQUIRE(res == SCP::EnvelopeState::INVALID); + SECTION("proceeds after value becomes validated") + { + auto envsBeforeClear = scp.mEnvs.size(); + + // Simulate tx set arrival — value becomes fully validated + scp.clearDownload(xValue); + + // Trigger advanceSlot with envelopes that confirm-prepare at a + // higher counter, so attemptConfirmPrepared finds newH > mHighBallot. + // This causes setConfirmPrepared to be called with the now-validated + // value, setting mCommit. + SCPBallot xB2(2, xValue); + REQUIRE(scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, xB2, &xB2)) == + SCP::EnvelopeState::VALID); + REQUIRE(scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, xB2, &xB2)) == + SCP::EnvelopeState::VALID); + + // setConfirmPrepared should now succeed — mCommit set, node + // progresses. Expect at least one new envelope with nC > 0 or a + // CONFIRM/EXTERNALIZE. + REQUIRE(scp.mEnvs.size() > envsBeforeClear); + bool foundC = false; + for (size_t i = envsBeforeClear; i < scp.mEnvs.size(); i++) + { + auto const& st = scp.mEnvs[i].statement; + if (st.pledges.type() == SCP_ST_PREPARE) + { + if (st.pledges.prepare().nC > 0) + { + foundC = true; + break; + } + } + else + { + // CONFIRM or EXTERNALIZE also proves we got past the stall + foundC = true; + break; + } + } + REQUIRE(foundC); + } } -TEST_CASE("drop tx set on download timeout", "[scp][ballotprotocol]") +TEST_CASE("incoming PREPARE with structurally valid prepared value is accepted", + "[scp][ballotprotocol]") { setupValues(); SIMULATION_CREATE_NODE(0); SIMULATION_CREATE_NODE(1); SIMULATION_CREATE_NODE(2); - // 3 node network with threshold=2 (need any 2 nodes to form quorum) SCPQuorumSet qSet; qSet.threshold = 2; qSet.validators.push_back(v0NodeID); @@ -3702,124 +3186,24 @@ TEST_CASE("drop tx set on download timeout", "[scp][ballotprotocol]") TestSCP scp(v0SecretKey.getPublicKey(), qSet); scp.storeQuorumSet(std::make_shared(qSet)); - uint256 qSetHash0 = scp.mSCP.getLocalNode()->getQuorumSetHash(); - SECTION("timeout during prepare phase") - { - // Node v0 starts ballot protocol with xValue - // Simulate that xValue is awaiting download with timeout exceeded - scp.startDownload(xValue, OVER_TX_SET_TIMEOUT); + // v0 enters ballot protocol with yValue + REQUIRE(scp.bumpState(0, yValue)); + REQUIRE(scp.mEnvs.size() == 1); - // Now call bumpState which should trigger - // maybeReplaceValueWithEmptyTxSet - REQUIRE(scp.bumpState(0, xValue)); - REQUIRE(scp.mEnvs.size() == 1); + // Set xValue to kStructurallyValidValue + scp.mValidateValueOverride = xValueStructurallyValidValidationOverride; - // The ballot should have an empty-tx-set value, not the original - // xValue - auto const& emittedBallot = - scp.mEnvs[0].statement.pledges.prepare().ballot; - - REQUIRE(emittedBallot.counter == 1); - REQUIRE(scp.isEmptyTxSetValue(emittedBallot.value)); - - // Verify it's the empty-tx-set value derived from xValue - Value expectedEmptyTxSetValue = - scp.makeEmptyTxSetValueFromValue(xValue); - REQUIRE(emittedBallot.value == expectedEmptyTxSetValue); - - verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, - SCPBallot(1, expectedEmptyTxSetValue)); - } - - SECTION("no timeout when wait time under threshold") - { - // Node v0 starts ballot protocol with xValue - REQUIRE(scp.bumpState(0, xValue)); - REQUIRE(scp.mEnvs.size() == 1); - - SCPBallot b1(1, xValue); - - // Simulate that xValue is awaiting download but wait time is still low - scp.startDownload(xValue, UNDER_TX_SET_TIMEOUT); - - // Try to bump state - should NOT replace with an empty-tx-set value - REQUIRE(scp.bumpState(0, xValue)); - REQUIRE(scp.mEnvs.size() == 2); - - // Verify ballot still has original xValue, not an empty-tx-set value - auto const& emittedBallot = - scp.mEnvs[1].statement.pledges.prepare().ballot; - REQUIRE(emittedBallot.counter == 2); - REQUIRE(!scp.isEmptyTxSetValue(emittedBallot.value)); - REQUIRE(emittedBallot.value == xValue); - } - - SECTION("empty-tx-set value can be prepared and confirmed") - { - // Start with xValue and timeout to an empty-tx-set value - scp.startDownload(xValue, OVER_TX_SET_TIMEOUT); - - REQUIRE(scp.bumpState(0, xValue)); - REQUIRE(scp.mEnvs.size() == 1); - - Value emptyTxSetValue = scp.makeEmptyTxSetValueFromValue(xValue); - SCPBallot emptyTxSetB1(1, emptyTxSetValue); - - // Verify we emitted an empty-tx-set value - REQUIRE(scp.isEmptyTxSetValue( - scp.mEnvs[0].statement.pledges.prepare().ballot.value)); - verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, emptyTxSetB1); - - // Other nodes also move to the empty-tx-set value - scp.receiveEnvelope( - makePrepare(v1SecretKey, qSetHash, 0, emptyTxSetB1)); - scp.receiveEnvelope( - makePrepare(v2SecretKey, qSetHash, 0, emptyTxSetB1)); - - // Should prepare the empty-tx-set value (quorum reached) - REQUIRE(scp.mEnvs.size() == 2); - verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, emptyTxSetB1, - &emptyTxSetB1); - - // Quorum confirms prepared empty-tx-set value - scp.receiveEnvelope( - makePrepare(v1SecretKey, qSetHash, 0, emptyTxSetB1, &emptyTxSetB1)); - scp.receiveEnvelope( - makePrepare(v2SecretKey, qSetHash, 0, emptyTxSetB1, &emptyTxSetB1)); - - REQUIRE(scp.mEnvs.size() == 3); - verifyPrepare(scp.mEnvs[2], v0SecretKey, qSetHash0, 0, emptyTxSetB1, - &emptyTxSetB1, 1, 1); - - // Accept commit - scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, emptyTxSetB1, - &emptyTxSetB1, 1, 1)); - scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, emptyTxSetB1, - &emptyTxSetB1, 1, 1)); - - REQUIRE(scp.mEnvs.size() == 4); - verifyConfirm(scp.mEnvs[3], v0SecretKey, qSetHash0, 0, 1, emptyTxSetB1, - 1, 1); - - // Externalize the empty-tx-set value - scp.receiveEnvelope( - makeConfirm(v1SecretKey, qSetHash, 0, 1, emptyTxSetB1, 1, 1)); - scp.receiveEnvelope( - makeConfirm(v2SecretKey, qSetHash, 0, 1, emptyTxSetB1, 1, 1)); - - REQUIRE(scp.mEnvs.size() == 5); - verifyExternalize(scp.mEnvs[4], v0SecretKey, qSetHash0, 0, emptyTxSetB1, - 1); - - // Verify the externalized value is the empty-tx-set value - REQUIRE(scp.mExternalizedValues.size() == 1); - REQUIRE(scp.isEmptyTxSetValue(scp.mExternalizedValues[0])); - REQUIRE(scp.mExternalizedValues[0] == emptyTxSetValue); - } + // v1 sends PREPARE with valid ballot value but only structurally valid + // prepared value. This should be accepted. + SCPBallot yB1(1, yValue); + SCPBallot xB1(1, xValue); + REQUIRE( + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, yB1, &xB1)) == + SCP::EnvelopeState::VALID); } -TEST_CASE("Proper handling of non-current ledger value", +TEST_CASE("incoming PREPARE with non-tx-set-invalid value is dropped", "[scp][ballotprotocol]") { setupValues(); @@ -3833,180 +3217,994 @@ TEST_CASE("Proper handling of non-current ledger value", qSet.validators.push_back(v1NodeID); qSet.validators.push_back(v2NodeID); + uint256 qSetHash = sha256(xdr::xdr_to_opaque(qSet)); + TestSCP scp(v0SecretKey.getPublicKey(), qSet); scp.storeQuorumSet(std::make_shared(qSet)); - // validateValue reports xValue is not for the current ledger. - scp.mValidateValueOverride = xValueNotCurrentLedgerOverride; + // xValue is invalid for some non-tx-set reason (close time, signature, + // ...). This should NOT be accepted as kStructurallyValidValue; the + // statement should be dropped. + scp.mValidateValueOverride = xValueNonTxSetInvalidValidationOverride; - REQUIRE(scp.bumpState(0, xValue)); + SCPBallot xB1(1, xValue); - // Do not emit a ballot with a kMaybeValid value - REQUIRE(scp.mEnvs.size() == 0); + // Envelope should be rejected as invalid + REQUIRE(scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1)) == + SCP::EnvelopeState::INVALID); + + // Envelope was not recorded — recordEnvelope only runs for non-kInvalid. + REQUIRE(scp.mSCP.getLatestMessage(v1NodeID) == nullptr); + // No local emit triggered. + REQUIRE(scp.mEnvs.empty()); } -TEST_CASE("setConfirmPrepared stalls on kStructurallyValidValue value", - "[scp][ballotprotocol]") +// --------------------------------------------------------------------------- +// Leader rounds +// +// There is no nomination phase. A round is a ballot counter; each round has +// one leader (SCPDriver::getRoundLeader). At counter r a node votes for the +// ballot `leader(r)` votes for at counter r, unless it already has a value it +// must stick to. +// --------------------------------------------------------------------------- + +namespace +{ +SCPBallot const& +latestBallot(TestSCP& scp) +{ + REQUIRE(!scp.mEnvs.empty()); + auto const& pl = scp.mEnvs.back().statement.pledges; + REQUIRE(pl.type() == SCP_ST_PREPARE); + return pl.prepare().ballot; +} +} + +TEST_CASE("leader rounds core5", "[scp][leaderschedule]") { setupValues(); SIMULATION_CREATE_NODE(0); SIMULATION_CREATE_NODE(1); SIMULATION_CREATE_NODE(2); + SIMULATION_CREATE_NODE(3); + SIMULATION_CREATE_NODE(4); + // we need 5 nodes to avoid sharing various thresholds: + // v-blocking set size: 2 + // threshold: 4 = 3 + self or 4 others SCPQuorumSet qSet; - qSet.threshold = 2; + qSet.threshold = 4; qSet.validators.push_back(v0NodeID); qSet.validators.push_back(v1NodeID); qSet.validators.push_back(v2NodeID); + qSet.validators.push_back(v3NodeID); + qSet.validators.push_back(v4NodeID); uint256 qSetHash = sha256(xdr::xdr_to_opaque(qSet)); TestSCP scp(v0SecretKey.getPublicKey(), qSet); scp.storeQuorumSet(std::make_shared(qSet)); + uint256 qSetHash0 = scp.mSCP.getLocalNode()->getQuorumSetHash(); - // Simulate parallel downloading - scp.startDownload(xValue, UNDER_TX_SET_TIMEOUT); + // round r is led by schedule[(r - 1) % size] + std::vector schedule; + scp.mRoundLeader = [&](uint64, uint32 round) -> std::optional { + if (schedule.empty()) + { + return std::nullopt; + } + return schedule[(round - 1) % schedule.size()]; + }; - // v0 enters ballot protocol - REQUIRE(scp.bumpState(0, xValue)); - REQUIRE(scp.mEnvs.size() == 1); - SCPBallot xB1(1, xValue); + SCPBallot x1(1, xValue), y1(1, yValue), x2(2, xValue), y2(2, yValue), + z2(2, zValue), z3(3, zValue); - // v1 and v2 send PREPAREs with prepared — quorum confirms prepared - REQUIRE( - scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1, &xB1)) == - SCP::EnvelopeState::VALID); - REQUIRE( - scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, xB1, &xB1)) == - SCP::EnvelopeState::VALID); + SECTION("the round 1 leader proposes when the slot starts") + { + schedule = {v0NodeID, v1NodeID, v2NodeID, v3NodeID, v4NodeID}; + REQUIRE(scp.startSlot(0, xValue)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1); + REQUIRE(scp.mValuesBuilt == 1); + REQUIRE(scp.getLeaderRound(0) == 1); + REQUIRE(scp.mProposedValues.size() == 1); + REQUIRE(scp.mProposedValues[0] == std::make_pair(uint32(1), xValue)); + // it has a ballot at its round: the ballot timer rules apply, not the + // leader timer + REQUIRE(!scp.hasLeaderTimer()); + + SECTION("starting again changes nothing") + { + REQUIRE(!scp.startSlot(0, yValue)); + REQUIRE(scp.mEnvs.size() == 1); + REQUIRE(scp.mValuesBuilt == 1); + } + SECTION("other nodes' ballots do not move the leader") + { + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, y1)); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, y1)); + REQUIRE(scp.mEnvs.size() == 1); + } + } - SECTION("commit gate stalls mCommit but mHighBallot is set") + SECTION("a follower waits for the leader and adopts its ballot") { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, yValue)); + REQUIRE(scp.mEnvs.empty()); + // followers do not build a value + REQUIRE(scp.mValuesBuilt == 0); + REQUIRE(scp.getLeaderRound(0) == 1); + REQUIRE(scp.hasLeaderTimer()); + + // a ballot from a node that does not lead the round is recorded but + // not adopted + REQUIRE( + scp.receiveEnvelope(makePrepare(v4SecretKey, qSetHash, 0, y1)) == + SCP::EnvelopeState::VALID); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.hasLeaderTimer()); - // setConfirmPrepared sets mHighBallot (nH > 0) but the commit gate - // stalls mCommit (nC == 0) because xValue is kStructurallyValidValue. - // Check the latest emitted PREPARE. - REQUIRE(scp.mEnvs.size() >= 2); - auto const& lastPrep = scp.mEnvs.back().statement.pledges.prepare(); - REQUIRE(lastPrep.nH == 1); - REQUIRE(lastPrep.nC == 0); + // the leader's ballot is adopted + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1); + REQUIRE(!scp.hasLeaderTimer()); + REQUIRE(scp.mValuesBuilt == 0); + + SECTION("and externalizes with a quorum") + { + // quorum votes prepare x1 -> accept prepared + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, x1)); + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 2); + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, x1, &x1); + // quorum accepted prepared -> confirm prepared, vote commit + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, x1, &x1)); + REQUIRE(scp.mEnvs.size() == 3); + verifyPrepare(scp.mEnvs[2], v0SecretKey, qSetHash0, 0, x1, &x1, 1, + 1); + // quorum votes commit -> accept commit + scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, x1, &x1, 1, 1)); + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, x1, &x1, 1, 1)); + scp.receiveEnvelope( + makePrepare(v3SecretKey, qSetHash, 0, x1, &x1, 1, 1)); + REQUIRE(scp.mEnvs.size() == 4); + verifyConfirm(scp.mEnvs[3], v0SecretKey, qSetHash0, 0, 1, x1, 1, 1); + // quorum accepted commit -> externalize + scp.receiveEnvelope( + makeConfirm(v1SecretKey, qSetHash, 0, 1, x1, 1, 1)); + scp.receiveEnvelope( + makeConfirm(v2SecretKey, qSetHash, 0, 1, x1, 1, 1)); + scp.receiveEnvelope( + makeConfirm(v3SecretKey, qSetHash, 0, 1, x1, 1, 1)); + REQUIRE(scp.mEnvs.size() == 5); + verifyExternalize(scp.mEnvs[4], v0SecretKey, qSetHash0, 0, x1, 1); + REQUIRE(scp.mExternalizedValues[0] == xValue); + REQUIRE(scp.mValuesBuilt == 0); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("a later ballot of the same leader at the same counter is " + "not adopted") + { + // equivocating leader: the node already voted for x1 + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, y1)); + REQUIRE(scp.mEnvs.size() == 1); + } } - SECTION("proceeds after value becomes validated") + SECTION("a follower adopts the round 1 leader before its own trigger") { - auto envsBeforeClear = scp.mEnvs.size(); + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(scp.getLeaderRound(0) == 0); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1); + REQUIRE(scp.getLeaderRound(0) == 1); - // Simulate tx set arrival — value becomes fully validated - scp.clearDownload(xValue); + // its own trigger then has nothing left to do + REQUIRE(!scp.startSlot(0, yValue)); + REQUIRE(scp.mEnvs.size() == 1); + REQUIRE(scp.mValuesBuilt == 0); + } - // Trigger advanceSlot with envelopes that confirm-prepare at a - // higher counter, so attemptConfirmPrepared finds newH > mHighBallot. - // This causes setConfirmPrepared to be called with the now-validated - // value, setting mCommit. - SCPBallot xB2(2, xValue); - REQUIRE(scp.receiveEnvelope( - makePrepare(v1SecretKey, qSetHash, 0, xB2, &xB2)) == - SCP::EnvelopeState::VALID); - REQUIRE(scp.receiveEnvelope( - makePrepare(v2SecretKey, qSetHash, 0, xB2, &xB2)) == - SCP::EnvelopeState::VALID); + SECTION("before its trigger a node does not adopt later rounds' leaders") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.getLeaderRound(0) == 0); + REQUIRE(!scp.hasLeaderTimer()); + } - // setConfirmPrepared should now succeed — mCommit set, node - // progresses. Expect at least one new envelope with nC > 0 or a - // CONFIRM/EXTERNALIZE. - REQUIRE(scp.mEnvs.size() > envsBeforeClear); - bool foundC = false; - for (size_t i = envsBeforeClear; i < scp.mEnvs.size(); i++) + SECTION("a leader that is not started does not propose") + { + schedule = {v0NodeID, v1NodeID, v2NodeID, v3NodeID, v4NodeID}; + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, y1)); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.mValuesBuilt == 0); + } + + SECTION("silent leader: the leader timer hands over to the next leader") + { + SECTION("local node leads round 2") { - auto const& st = scp.mEnvs[i].statement; - if (st.pledges.type() == SCP_ST_PREPARE) + schedule = {v1NodeID, v0NodeID, v2NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, yValue)); + REQUIRE(scp.hasLeaderTimer()); + auto timer = scp.getLeaderTimer(); + REQUIRE(timer.mAbsoluteTimeout == + std::chrono::milliseconds(scp.mInitialBallotTimeoutMS)); + + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnteredRounds.back() == + std::make_pair(uint32(2), true)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, y2); + REQUIRE(scp.mValuesBuilt == 1); + REQUIRE(!scp.hasLeaderTimer()); + + SECTION("the round 1 leader showing up late changes nothing") { - if (st.pledges.prepare().nC > 0) - { - foundC = true; - break; - } + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); } - else + SECTION("a stale leader timer callback is ignored") { - // CONFIRM or EXTERNALIZE also proves we got past the stall - foundC = true; - break; + timer.mCallback(); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnvs.size() == 1); } } - REQUIRE(foundC); - } -} + SECTION("another node leads round 2") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, zValue)); -TEST_CASE("incoming PREPARE with structurally valid prepared value is accepted", - "[scp][ballotprotocol]") -{ - setupValues(); - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); + SECTION("its ballot arrives after the timeout") + { + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnvs.empty()); + // waiting for the round 2 leader now, with the round 2 + // timeout + REQUIRE(scp.hasLeaderTimer()); + REQUIRE( + scp.getLeaderTimer().mAbsoluteTimeout == + std::chrono::milliseconds(scp.mInitialBallotTimeoutMS + + scp.mIncrementBallotTimeoutMS)); + + SECTION("the round 2 leader's ballot is adopted") + { + // a round 2 ballot from a node that does not lead round 2 + scp.receiveEnvelope( + makePrepare(v3SecretKey, qSetHash, 0, x2)); + REQUIRE(scp.mEnvs.empty()); - SCPQuorumSet qSet; - qSet.threshold = 2; - qSet.validators.push_back(v0NodeID); - qSet.validators.push_back(v1NodeID); - qSet.validators.push_back(v2NodeID); + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, y2); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("a late round 1 proposal still gets the node's vote") + { + // The node never voted: it votes for the round 1 leader's + // late proposal at the round 1 counter, and keeps waiting + // for the round 2 leader. + scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.hasLeaderTimer()); + REQUIRE(!scp.hasBallotTimer()); - uint256 qSetHash = sha256(xdr::xdr_to_opaque(qSet)); + SECTION("and externalizes with a quorum at round 1") + { + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope( + makePrepare(v3SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope( + makePrepare(v4SecretKey, qSetHash, 0, x1, &x1)); + // accepted then confirmed prepared: the node votes to + // commit x at counter 1 + REQUIRE(scp.mEnvs.size() >= 2); + auto const& last = + scp.mEnvs.back().statement.pledges.prepare(); + REQUIRE(last.ballot.value == xValue); + REQUIRE(last.nC == 1); + REQUIRE(last.nH == 1); + } + SECTION("the round 2 leader's ballot takes over") + { + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 2); + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, + y2); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("a second late proposal is ignored") + { + // only a node without a ballot takes a late proposal + schedule = {v1NodeID, v2NodeID, v3NodeID, v0NodeID, + v4NodeID}; + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 3); + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 1); + } + } + SECTION("a proposal that is not valid is not taken late either") + { + scp.mValidateValueOverride = [&](uint64, Value const& v, + bool proposal) { + return (proposal && v == xValue) + ? SCPDriver::kInvalidValue + : SCPDriver::kFullyValidatedValue; + }; + scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.empty()); + } + } + SECTION("the round 1 leader's proposal is pending: one extension") + { + // The application holds the leader's statement (e.g. it is + // fetching its transaction set): the node waits one more + // timeout, once. + scp.mProposalPending = true; + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 1); + REQUIRE(scp.hasLeaderTimer()); + REQUIRE(scp.getLeaderTimer().mAbsoluteTimeout == + std::chrono::milliseconds(scp.mInitialBallotTimeoutMS)); + REQUIRE(scp.mEnvs.empty()); + + SECTION("the proposal arrives during the extension") + { + scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1); + REQUIRE(scp.getLeaderRound(0) == 1); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("it does not: the node moves on") + { + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.hasLeaderTimer()); + REQUIRE(scp.mEnvs.empty()); + } + } + SECTION("its ballot arrived early: kept until the round starts") + { + // a single node cannot make others skip a round + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.getLeaderRound(0) == 1); - TestSCP scp(v0SecretKey.getPublicKey(), qSet); - scp.storeQuorumSet(std::make_shared(qSet)); + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, y2); + } + SECTION("then local node leads round 3 with its own value") + { + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.mEnvs.empty()); + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 3); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, z3); + REQUIRE(scp.mValuesBuilt == 1); + } + } + } - // v0 enters ballot protocol with yValue - REQUIRE(scp.bumpState(0, yValue)); - REQUIRE(scp.mEnvs.size() == 1); + SECTION("ballot timeout: the next leader takes over") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, zValue)); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); - // Set xValue to kStructurallyValidValue - scp.mValidateValueOverride = xValueStructurallyValidValidationOverride; + // a quorum is at counter 1: the ballot timer runs + REQUIRE(!scp.hasBallotTimer()); + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, y1)); + scp.receiveEnvelope(makePrepare(v4SecretKey, qSetHash, 0, y1)); + REQUIRE(scp.hasBallotTimer()); + REQUIRE(!scp.hasLeaderTimer()); - // v1 sends PREPARE with valid ballot value but only structurally valid - // prepared value. This should be accepted. - SCPBallot yB1(1, yValue); - SCPBallot xB1(1, xValue); - REQUIRE( - scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, yB1, &xB1)) == - SCP::EnvelopeState::VALID); + SECTION("to a leader whose ballot is known") + { + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 1); + scp.fireTimer(Slot::BALLOT_PROTOCOL_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnvs.size() == 2); + // v3 and v4 voted for y at counter 1: with the leader and the + // local node that is a quorum, so (1, y) is accepted prepared + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, y2, &y1); + } + SECTION("to a leader that has not spoken yet") + { + scp.fireTimer(Slot::BALLOT_PROTOCOL_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + // keeps its round 1 ballot, votes for nothing new + REQUIRE(scp.mEnvs.size() == 1); + REQUIRE(scp.hasLeaderTimer()); + // the ballot timer of the round it gave up on is stopped + REQUIRE(!scp.hasBallotTimer()); + + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 2); + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, y2, &y1); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("to the local node, which keeps the value it voted for") + { + schedule = {v1NodeID, v0NodeID, v2NodeID, v3NodeID, v4NodeID}; + scp.fireTimer(Slot::BALLOT_PROTOCOL_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnvs.size() == 2); + // x, not its own z: the transaction set of x is already out + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, x2); + REQUIRE(scp.mValuesBuilt == 0); + } + SECTION("to the local node, whose current value is not proposable") + { + schedule = {v1NodeID, v0NodeID, v2NodeID, v3NodeID, v4NodeID}; + scp.mValidateValueOverride = [&](uint64, Value const& v, + bool proposal) { + return (proposal && v == xValue) + ? SCPDriver::kInvalidValue + : SCPDriver::kFullyValidatedValue; + }; + scp.fireTimer(Slot::BALLOT_PROTOCOL_TIMER); + REQUIRE(scp.mEnvs.size() == 2); + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, z2); + REQUIRE(scp.mValuesBuilt == 1); + } + } + + SECTION("a confirmed prepared value wins over the leader's") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, zValue)); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, x1)); + scp.receiveEnvelope(makePrepare(v4SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 2); + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, x1, &x1); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope(makePrepare(v4SecretKey, qSetHash, 0, x1, &x1)); + // confirmed prepared: x is now the value to stick to + REQUIRE(scp.mEnvs.size() == 3); + verifyPrepare(scp.mEnvs[2], v0SecretKey, qSetHash0, 0, x1, &x1, 1, 1); + + // the round 2 leader votes for something else + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 3); + scp.fireTimer(Slot::BALLOT_PROTOCOL_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnvs.size() == 4); + verifyPrepare(scp.mEnvs[3], v0SecretKey, qSetHash0, 0, x2, &x1, 1, 1); + REQUIRE(!scp.hasLeaderTimer()); + } + + SECTION("a v-blocking set ahead moves a node to that round") + { + schedule = {v1NodeID, v2NodeID, v3NodeID, v0NodeID, v4NodeID}; + // v1 and v2 (v-blocking) are at counter 3, on a value they must + // stick to; local node has no ballot at all + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, z3)); + REQUIRE(scp.getLeaderRound(0) == 0); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, z3)); + REQUIRE(scp.getLeaderRound(0) == 3); + // it does not vote for what they vote for: only the round's leader + // decides + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.hasLeaderTimer()); + + SECTION("where it adopts the leader") + { + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, z3)); + REQUIRE(scp.mEnvs.size() == 1); + // with v1, v2 and the leader that is a quorum voting for z3 + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, z3, &z3); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("where it leads once it is started") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(scp.startSlot(0, xValue)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, + SCPBallot(3, xValue)); + } + } + + SECTION("a started leader pushed to its round by a v-blocking set " + "proposes") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, xValue)); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, z3)); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, z3)); + REQUIRE(scp.getLeaderRound(0) == 3); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, + SCPBallot(3, xValue)); + REQUIRE(scp.mValuesBuilt == 1); + } + + SECTION("proposal validation gates adoption") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, zValue)); + + SECTION("a value the node would not propose is not adopted") + { + // valid for the ballot protocol, but not as a proposal (e.g. it + // carries an upgrade the local node did not arm) + scp.mValidateValueOverride = [&](uint64, Value const& v, + bool proposal) { + return (proposal && v == xValue) + ? SCPDriver::kInvalidValue + : SCPDriver::kFullyValidatedValue; + }; + REQUIRE(scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, x1)) == + SCP::EnvelopeState::VALID); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.hasLeaderTimer()); + + // the regular protocol still applies: a quorum that prepared the + // value brings the node along + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, x1, &x1)); + // v-blocking accepted prepared + REQUIRE(scp.mEnvs.empty()); + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope(makePrepare(v4SecretKey, qSetHash, 0, x1, &x1)); + // quorum accepted prepared -> confirmed prepared + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1, &x1, 1, + 1); + } + SECTION("a value whose transaction set is still downloading is " + "adopted") + { + scp.startDownload(xValue, UNDER_TX_SET_TIMEOUT); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1); + } + } + + SECTION("a leader that cannot build a value lets the round time out") + { + schedule = {v0NodeID, v1NodeID, v2NodeID, v3NodeID, v4NodeID}; + scp.mMakeValueOverride = []() { return ValueWrapperPtr(); }; + REQUIRE(!scp.startSlot(0, xValue)); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.mValuesBuilt == 1); + REQUIRE(scp.hasLeaderTimer()); + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, y2); + } + + SECTION("building the value re-enters SCP") + { + schedule = {v0NodeID, v1NodeID, v2NodeID, v3NodeID, v4NodeID}; + auto xW = scp.wrapValue(xValue); + + SECTION("the slot moved to a later round meanwhile") + { + scp.mMakeValueOverride = [&]() { + // envelopes that became ready while the value was built + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, z3)); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, z3)); + return xW; + }; + REQUIRE(scp.startSlot(0, xValue)); + // the value was built for round 1; the node is in round 3 now, + // which v2 leads: it votes for v2's ballot, not for its own value + REQUIRE(scp.getLeaderRound(0) == 3); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, z3); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("the slot externalized meanwhile") + { + scp.mMakeValueOverride = [&]() { + for (auto const& k : + {v1SecretKey, v2SecretKey, v3SecretKey, v4SecretKey}) + { + scp.receiveEnvelope(makeExternalize(k, qSetHash, 0, y1, 1)); + } + return xW; + }; + scp.startSlot(0, xValue); + REQUIRE(scp.mExternalizedValues[0] == yValue); + for (auto const& e : scp.mEnvs) + { + // never voted for its own value + auto vals = Slot::getStatementValues(e.statement); + REQUIRE(std::find(vals.begin(), vals.end(), xValue) == + vals.end()); + } + REQUIRE(!scp.hasLeaderTimer()); + } + } + + SECTION("no schedule known: neither proposes nor adopts") + { + // e.g. the slot is not the next ledger to close + REQUIRE(!scp.startSlot(0, xValue)); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.mValuesBuilt == 0); + } + + SECTION("non-validators never vote for a leader's ballot") + { + TestSCP watcher(v0SecretKey.getPublicKey(), qSet, false); + watcher.storeQuorumSet(std::make_shared(qSet)); + watcher.mRoundLeader = [&](uint64, uint32) { return v1NodeID; }; + watcher.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(watcher.mEnvs.empty()); + REQUIRE(watcher.getLeaderRound(0) == 0); + REQUIRE(!watcher.hasLeaderTimer()); + } + + SECTION("nomination statements are dropped") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(scp.receiveEnvelope( + makeNominate(v1SecretKey, qSetHash, 0, {xValue}, {})) == + SCP::EnvelopeState::INVALID); + REQUIRE(scp.mSCP.getLatestMessage(v1NodeID) == nullptr); + REQUIRE(scp.mEnvs.empty()); + } } -TEST_CASE("incoming PREPARE with non-tx-set-invalid value is dropped", - "[scp][ballotprotocol]") +namespace { - setupValues(); - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); +// A small network of TestSCP nodes on one slot, driven by a virtual clock: +// messages are delayed, reordered and (for a while) lost; some nodes are dead; +// nodes start the slot at different times. +struct LeaderRoundsModel +{ + struct Node + { + SecretKey mKey; + std::unique_ptr mSCP; + bool mAlive{true}; + bool mStarted{false}; + std::chrono::milliseconds mStartTime{0}; + size_t mEnvsSeen{0}; + Value mValue; + }; - SCPQuorumSet qSet; - qSet.threshold = 2; - qSet.validators.push_back(v0NodeID); - qSet.validators.push_back(v1NodeID); - qSet.validators.push_back(v2NodeID); + struct Delivery + { + std::chrono::milliseconds mWhen; + uint64_t mSeq; + size_t mTo; + SCPEnvelope mEnv; + bool + operator>(Delivery const& o) const + { + return std::tie(mWhen, mSeq) > std::tie(o.mWhen, o.mSeq); + } + }; - uint256 qSetHash = sha256(xdr::xdr_to_opaque(qSet)); + stellar_default_random_engine mRng; + std::vector mNodes; + std::vector mSchedule; + std::priority_queue, std::greater<>> mQueue; + uint64_t mSeq{0}; + std::chrono::milliseconds mNow{0}; + // until then, messages are lost with probability mLossPercent and can be + // delayed up to mMaxChaosDelay + std::chrono::milliseconds mChaosEnd{0}; + uint32_t mLossPercent{0}; + std::chrono::milliseconds mMaxChaosDelay{0}; + std::chrono::milliseconds mNextRebroadcast{0}; + size_t mThreshold{0}; + + LeaderRoundsModel(uint32_t seed, size_t numNodes, size_t numDead) + : mRng(seed) + { + mThreshold = numNodes - (numNodes - 1) / 3; + releaseAssert(numNodes - numDead >= mThreshold); + SCPQuorumSet qSet; + qSet.threshold = static_cast(mThreshold); + for (size_t i = 0; i < numNodes; i++) + { + Node n; + n.mKey = SecretKey::fromSeed( + sha256(fmt::format("LEADER_MODEL_NODE_{}", i))); + qSet.validators.push_back(n.mKey.getPublicKey()); + // distinct proposals + n.mValue = xdr::xdr_to_opaque( + sha256(fmt::format("LEADER_MODEL_VALUE_{}_{}", seed, i))); + mNodes.emplace_back(std::move(n)); + } + for (auto const& n : mNodes) + { + mSchedule.emplace_back(n.mKey.getPublicKey()); + } + stellar::shuffle(mSchedule.begin(), mSchedule.end(), mRng); - TestSCP scp(v0SecretKey.getPublicKey(), qSet); - scp.storeQuorumSet(std::make_shared(qSet)); + // dead nodes are picked from the front of the schedule half of the + // time (so the first leaders are dead), at random otherwise + std::vector order(numNodes); + std::iota(order.begin(), order.end(), 0); + if (rand_uniform(0, 1, mRng) == 0) + { + std::sort(order.begin(), order.end(), [&](size_t a, size_t b) { + auto pos = [&](size_t i) { + return std::find(mSchedule.begin(), mSchedule.end(), + mNodes[i].mKey.getPublicKey()) - + mSchedule.begin(); + }; + return pos(a) < pos(b); + }); + } + else + { + stellar::shuffle(order.begin(), order.end(), mRng); + } + for (size_t i = 0; i < numDead; i++) + { + mNodes[order[i]].mAlive = false; + } - // xValue is invalid for some non-tx-set reason (close time, signature, - // ...). This should NOT be accepted as kStructurallyValidValue; the - // statement should be dropped. - scp.mValidateValueOverride = xValueNonTxSetInvalidValidationOverride; + mChaosEnd = + std::chrono::milliseconds(rand_uniform(0, 30000, mRng)); + mLossPercent = rand_uniform(0, 40, mRng); + mMaxChaosDelay = + std::chrono::milliseconds(rand_uniform(10, 4000, mRng)); + mNextRebroadcast = mChaosEnd; - SCPBallot xB1(1, xValue); + for (auto& n : mNodes) + { + n.mSCP = std::make_unique(n.mKey.getPublicKey(), qSet); + n.mSCP->mRoundLeader = [this](uint64, uint32 round) { + return mSchedule[(round - 1) % mSchedule.size()]; + }; + // most nodes start close together; some are late + n.mStartTime = std::chrono::milliseconds( + rand_uniform(0, 3, mRng) == 0 + ? rand_uniform(0, 8000, mRng) + : rand_uniform(0, 300, mRng)); + } + } - // Envelope should be rejected as invalid - REQUIRE(scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1)) == - SCP::EnvelopeState::INVALID); + void + send(size_t from, SCPEnvelope const& env) + { + for (size_t to = 0; to < mNodes.size(); to++) + { + if (to == from || !mNodes[to].mAlive) + { + continue; + } + bool chaos = mNow < mChaosEnd; + if (chaos && rand_uniform(0, 99, mRng) < mLossPercent) + { + continue; + } + auto maxDelay = + chaos ? mMaxChaosDelay : std::chrono::milliseconds(150); + auto delay = std::chrono::milliseconds( + rand_uniform(1, static_cast(maxDelay.count()), mRng)); + mQueue.push(Delivery{mNow + delay, mSeq++, to, env}); + } + } - // Envelope was not recorded — recordEnvelope only runs for non-kInvalid. - REQUIRE(scp.mSCP.getLatestMessage(v1NodeID) == nullptr); - // No local emit triggered. - REQUIRE(scp.mEnvs.empty()); + // forwards whatever the nodes emitted since the last call + void + flush() + { + for (size_t i = 0; i < mNodes.size(); i++) + { + auto& n = mNodes[i]; + while (n.mEnvsSeen < n.mSCP->mEnvs.size()) + { + auto env = n.mSCP->mEnvs[n.mEnvsSeen++]; + send(i, env); + } + } + } + + bool + allExternalized() const + { + for (auto const& n : mNodes) + { + if (n.mAlive && n.mSCP->mExternalizedValues.empty()) + { + return false; + } + } + return true; + } + + // runs until every live node externalized or `limit` (virtual time); + // returns true in the first case + bool + run(std::chrono::milliseconds limit) + { + while (!allExternalized()) + { + // next event: a start, a delivery, a timer or a rebroadcast + enum class Kind + { + NONE, + START, + DELIVERY, + TIMER, + REBROADCAST + }; + Kind kind = Kind::REBROADCAST; + auto when = mNextRebroadcast; + size_t node = 0; + int timerID = 0; + for (size_t i = 0; i < mNodes.size(); i++) + { + auto& n = mNodes[i]; + if (!n.mAlive) + { + continue; + } + if (!n.mStarted && n.mStartTime < when) + { + kind = Kind::START; + when = n.mStartTime; + node = i; + } + for (auto const& t : n.mSCP->mTimers) + { + if (t.second.mCallback && t.second.mAbsoluteTimeout < when) + { + kind = Kind::TIMER; + when = t.second.mAbsoluteTimeout; + node = i; + timerID = t.first; + } + } + } + if (!mQueue.empty() && mQueue.top().mWhen < when) + { + kind = Kind::DELIVERY; + when = mQueue.top().mWhen; + } + if (when > limit) + { + return false; + } + mNow = std::max(mNow, when); + for (auto& n : mNodes) + { + n.mSCP->mCurrentTimerOffset = mNow; + } + switch (kind) + { + case Kind::START: + { + auto& n = mNodes[node]; + n.mStarted = true; + n.mSCP->startSlot(0, n.mValue); + } + break; + case Kind::TIMER: + { + auto& td = mNodes[node].mSCP->mTimers[timerID]; + auto cb = td.mCallback; + td.mCallback = nullptr; + cb(); + } + break; + case Kind::DELIVERY: + { + auto d = mQueue.top(); + mQueue.pop(); + mNodes[d.mTo].mSCP->receiveEnvelope(d.mEnv); + } + break; + case Kind::REBROADCAST: + { + // like Herder, nodes periodically re-send their latest + // statement + mNextRebroadcast = mNow + std::chrono::milliseconds(2000); + for (size_t i = 0; i < mNodes.size(); i++) + { + auto& n = mNodes[i]; + if (n.mAlive && !n.mSCP->mEnvs.empty()) + { + send(i, n.mSCP->mEnvs.back()); + } + } + } + break; + default: + releaseAssert(false); + } + flush(); + } + return true; + } + + void + checkAgreement() const + { + std::optional externalized; + for (auto const& n : mNodes) + { + auto const& ext = n.mSCP->mExternalizedValues; + if (ext.empty()) + { + continue; + } + auto const& v = ext.begin()->second; + if (!externalized) + { + externalized = v; + // it is one of the proposals + REQUIRE( + std::any_of(mNodes.begin(), mNodes.end(), + [&](Node const& o) { return o.mValue == v; })); + } + REQUIRE(*externalized == v); + } + } +}; + +void +runLeaderRoundsModel(uint32_t firstSeed, uint32_t numSeeds) +{ + for (uint32_t seed = firstSeed; seed < firstSeed + numSeeds; seed++) + { + size_t const numNodes = 4 + seed % 4; // 4..7 + size_t const maxDead = (numNodes - 1) / 3; // 1, 1, 1, 2 + size_t const numDead = (seed / 4) % (maxDead + 1); + INFO(fmt::format("seed {} nodes {} dead {}", seed, numNodes, numDead)); + LeaderRoundsModel model(seed, numNodes, numDead); + // Once messages flow again, the slot must close: the bound leaves + // room for a full turn of the schedule with growing timeouts. + bool done = model.run(model.mChaosEnd + std::chrono::minutes(10)); + // safety holds no matter what + model.checkAgreement(); + REQUIRE(done); + } +} +} + +TEST_CASE("leader rounds network model", "[scp][leaderschedule]") +{ + runLeaderRoundsModel(1, 400); +} + +TEST_CASE("leader rounds network model long", "[scp][leaderschedule][!hide]") +{ + runLeaderRoundsModel(100000, 20000); } } diff --git a/src/scp/test/SCPUnitTests.cpp b/src/scp/test/SCPUnitTests.cpp deleted file mode 100644 index 50dbd484af..0000000000 --- a/src/scp/test/SCPUnitTests.cpp +++ /dev/null @@ -1,840 +0,0 @@ -#include "scp/LocalNode.h" -#include "scp/SCP.h" -#include "scp/Slot.h" -#include "simulation/Simulation.h" -#include "test/Catch2.h" -#include "util/Logging.h" -#include "xdrpp/marshal.h" - -namespace stellar -{ -static bool -isNear(uint64 r, double target) -{ - double v = (double)r / (double)UINT64_MAX; - return (std::abs(v - target) < .01); -} - -class TestNominationSCP : public SCPDriver -{ - public: - SCP mSCP; - uint32_t mInitialNominationTimeoutMS = 1000; - uint32_t mIncrementNominationTimeoutMS = 1000; - uint32_t mInitialBallotTimeoutMS = 1000; - uint32_t mIncrementBallotTimeoutMS = 1000; - - TestNominationSCP(NodeID const& nodeID, SCPQuorumSet const& qSetLocal) - : mSCP(*this, nodeID, true, qSetLocal) - { - auto localQSet = - std::make_shared(mSCP.getLocalQuorumSet()); - storeQuorumSet(localQSet); - } - - void - signEnvelope(SCPEnvelope&) override - { - } - - void - storeQuorumSet(SCPQuorumSetPtr qSet) - { - Hash qSetHash = sha256(xdr::xdr_to_opaque(*qSet.get())); - mQuorumSets[qSetHash] = qSet; - } - - SCPDriver::ValidationLevel - validateValue(uint64 slotIndex, Value const& value, - bool nomination) const override - { - return SCPDriver::kFullyValidatedValue; - } - - SCPQuorumSetPtr - getQSet(Hash const& qSetHash) override - { - if (mQuorumSets.find(qSetHash) != mQuorumSets.end()) - { - - return mQuorumSets[qSetHash]; - } - return SCPQuorumSetPtr(); - } - - void - emitEnvelope(SCPEnvelope const& envelope) override - { - } - - ValueWrapperPtr - combineCandidates(uint64 slotIndex, - ValueWrapperPtrSet const& candidates) override - { - return nullptr; - } - - bool - hasUpgrades(Value const& v) override - { - // Not implemented - releaseAssert(false); - } - - ValueWrapperPtr - stripAllUpgrades(Value const& v) override - { - // Not implemented - releaseAssert(false); - } - - uint32_t - getUpgradeNominationTimeoutLimit() const override - { - return std::numeric_limits::max(); - } - - void - setupTimer(uint64 slotIndex, int timerID, std::chrono::milliseconds timeout, - std::function cb) override - { - } - - void - stopTimer(uint64 slotIndex, int timerID) override - { - } - - std::optional - getTxSetDownloadWaitTime(Value const& v) const override - { - return std::nullopt; - } - - std::chrono::milliseconds - getTxSetDownloadTimeout() const override - { - return std::chrono::milliseconds(100); - } - - Value - makeEmptyTxSetValueFromValue(Value const& value) const override - { - releaseAssert(false); - } - - bool - isEmptyTxSetValue(Value const& v) const override - { - releaseAssert(false); - } - - bool - isParallelTxSetDownloadEnabled() const override - { - return true; - } - - bool - protocolAllowsEmptyTxSetValues() const override - { - return true; - } - - std::map mQuorumSets; - - Value const& - getLatestCompositeCandidate(uint64 slotIndex) - { - static Value const emptyValue{}; - return emptyValue; - } - - Hash - getHashOf(std::vector> const& vals) const override - { - SHA256 hasher; - for (auto const& v : vals) - { - hasher.add(v); - } - return hasher.finish(); - } - - // Copied from HerderSCPDriver.cpp - static uint32_t const MAX_TIMEOUT_MS = (30 * 60) * 1000; - - std::chrono::milliseconds - computeTimeout(uint32 roundNumber, bool isNomination) override - { - int initialTimeoutMS; - int incrementMS; - - if (isNomination) - { - initialTimeoutMS = mInitialNominationTimeoutMS; - incrementMS = mIncrementNominationTimeoutMS; - } - else - { - initialTimeoutMS = mInitialBallotTimeoutMS; - incrementMS = mIncrementBallotTimeoutMS; - } - - int timeoutMS = initialTimeoutMS + (roundNumber - 1) * incrementMS; - if (timeoutMS > MAX_TIMEOUT_MS) - { - timeoutMS = MAX_TIMEOUT_MS; - } - return std::chrono::milliseconds(timeoutMS); - } - - bool - isEnvelopeReady(SCPEnvelope const& envelope) const override - { - // Not implemented. These tests do not use PendingEnvelopes, and so do - // not require this method. - releaseAssert(false); - } -}; - -// Deliberately repeat the first winner in round 2, so the second call must -// fast-forward to round 3. This catches predicting round 2's hash in isolation. -class LeaderPreviewTestDriver : public TestNominationSCP -{ - public: - std::vector nodes; - bool tie = false; - std::set zeroWeight; - size_t emitted = 0; - size_t timers = 0; - std::function nominationTimeout; - - LeaderPreviewTestDriver(NodeID const& local, SCPQuorumSet const& qset) - : TestNominationSCP(local, qset), nodes(qset.validators) - { - } - - uint64 - getNodeWeight(NodeID const& id, SCPQuorumSet const&, bool) const override - { - return zeroWeight.count(id) ? 0 : UINT64_MAX; - } - - uint64 - computeHashNode(uint64, Value const&, bool priority, int32 round, - NodeID const& id) override - { - if (!priority) - { - return 0; - } - auto winner = round < 3 ? nodes[0] : round == 3 ? nodes[1] : nodes[2]; - return id == winner || (tie && round < 3 && id == nodes[2]) ? 100 : 1; - } - - void - emitEnvelope(SCPEnvelope const&) override - { - ++emitted; - } - - void - setupTimer(uint64, int timerID, std::chrono::milliseconds, - std::function cb) override - { - ++timers; - if (timerID == Slot::NOMINATION_TIMER) - { - nominationTimeout = std::move(cb); - } - } -}; - -TEST_CASE("nomination leader preview has no protocol side effects", - "[scp][early-nomination]") -{ - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - SCPQuorumSet qset; - qset.threshold = 2; - qset.validators = {v0NodeID, v1NodeID, v2NodeID}; - // The local node is the *second* leader. Merely preparing its value must - // not cause it to vote before the actual nomination timeout. - LeaderPreviewTestDriver driver(v1NodeID, qset); - auto& scp = driver.mSCP; - Value previous{42}, value{43}; - auto slot = std::make_shared(7, scp); - auto const first = scp.predictNominationLeaders(7, previous, 1); - auto const firstTwo = scp.predictNominationLeaders(7, previous, 2); - REQUIRE(first == std::set{v0NodeID}); - REQUIRE(firstTwo == std::set{v0NodeID, v1NodeID}); - REQUIRE(scp.getKnownSlotsCount() == 0); - REQUIRE(slot->getNominationLeaders().empty()); - REQUIRE(driver.emitted == 0); - REQUIRE(driver.timers == 0); - - size_t builds = 0; - slot->nominate( - [&]() { - ++builds; - return std::make_shared(value); - }, - previous, false); - REQUIRE(builds == 0); - REQUIRE(slot->getNominationLeaders() == first); - REQUIRE(driver.emitted == 0); - REQUIRE(driver.timers == 1); - // Preview again after nomination has started: it still changes no state. - REQUIRE(scp.predictNominationLeaders(7, previous, 2) == firstTwo); - REQUIRE(slot->getNominationLeaders() == first); - REQUIRE(driver.timers == 1); - auto timeout = std::move(driver.nominationTimeout); - REQUIRE(timeout); - timeout(); - REQUIRE(slot->getNominationLeaders() == firstTwo); - REQUIRE(driver.emitted == 1); - REQUIRE(driver.timers == 2); - REQUIRE(builds == 1); - timeout = std::move(driver.nominationTimeout); - REQUIRE(timeout); - timeout(); - REQUIRE(builds == 1); -} - -TEST_CASE("lazy nomination follows leaders and handles unavailable values", - "[scp][early-nomination]") -{ - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - SCPQuorumSet qset; - qset.threshold = 3; - qset.validators = {v0NodeID, v1NodeID, v2NodeID}; - LeaderPreviewTestDriver driver(v1NodeID, qset); - auto& scp = driver.mSCP; - Value previous{42}, value{43}; - size_t builds = 0; - bool available = true; - bool stopWhileBuilding = false; - NominationValueSupplier makeValue = [&]() -> ValueWrapperPtr { - ++builds; - if (stopWhileBuilding) - { - scp.stopNomination(7); - } - return available ? std::make_shared(value) : nullptr; - }; - - SECTION("a follower votes for the leader without constructing a value") - { - SCPEnvelope envelope; - envelope.statement.nodeID = v0NodeID; - envelope.statement.slotIndex = 7; - envelope.statement.pledges.type(SCP_ST_NOMINATE); - auto& nom = envelope.statement.pledges.nominate(); - nom.quorumSetHash = sha256(xdr::xdr_to_opaque(qset)); - nom.votes.push_back(value); - REQUIRE(scp.receiveEnvelope(driver.wrapEnvelope(envelope)) == - SCP::VALID); - REQUIRE(scp.nominate(7, makeValue, previous)); - REQUIRE(builds == 0); - REQUIRE(driver.emitted == 1); - auto timeout = std::move(driver.nominationTimeout); - timeout(); - REQUIRE(scp.getNominationLeaders(7).count(v1NodeID)); - // Even a promoted leader needs no private value if it already follows - // a usable value from the first leader. - REQUIRE(builds == 0); - } - SECTION("an unavailable proposal is retried on the next timeout") - { - available = false; - REQUIRE(!scp.nominate(7, makeValue, previous)); - REQUIRE(builds == 0); - auto timeout = std::move(driver.nominationTimeout); - timeout(); - REQUIRE(builds == 1); - REQUIRE(driver.emitted == 0); - available = true; - timeout = std::move(driver.nominationTimeout); - REQUIRE(timeout); - timeout(); - REQUIRE(builds == 2); - REQUIRE(driver.emitted == 1); - } - SECTION("stopping nomination during construction cannot emit or rearm") - { - stopWhileBuilding = true; - scp.nominate(7, makeValue, previous); - REQUIRE(builds == 0); - auto timeout = std::move(driver.nominationTimeout); - timeout(); - REQUIRE(builds == 1); - REQUIRE(driver.emitted == 0); - REQUIRE(driver.timers == 1); - } - SECTION("a stopped follower never constructs a proposal") - { - scp.nominate(7, makeValue, previous); - scp.stopNomination(7); - auto timeout = std::move(driver.nominationTimeout); - timeout(); - REQUIRE(builds == 0); - REQUIRE(driver.emitted == 0); - REQUIRE(driver.timers == 1); - } -} - -TEST_CASE("nomination leader preview preserves ties and zero weights", - "[scp][early-nomination]") -{ - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - SCPQuorumSet qset; - qset.threshold = 2; - qset.validators = {v0NodeID, v1NodeID, v2NodeID}; - LeaderPreviewTestDriver driver(v1NodeID, qset); - Value previous{42}; - SECTION("ties may elect more than two nodes in the first two calls") - { - driver.tie = true; - REQUIRE(driver.mSCP.predictNominationLeaders(7, previous, 1) == - std::set{v0NodeID, v2NodeID}); - REQUIRE(driver.mSCP.predictNominationLeaders(7, previous, 2) == - std::set{v0NodeID, v1NodeID, v2NodeID}); - } - SECTION("zero weight winners are excluded, including self") - { - driver.zeroWeight = {v0NodeID, v1NodeID}; - REQUIRE(driver.mSCP.predictNominationLeaders(7, previous, 2) == - std::set{v2NodeID}); - } - SECTION("no eligible nodes") - { - driver.zeroWeight = {v0NodeID, v1NodeID, v2NodeID}; - REQUIRE(driver.mSCP.predictNominationLeaders(7, previous, 2).empty()); - } - REQUIRE(driver.emitted == 0); - REQUIRE(driver.timers == 0); - REQUIRE(driver.mSCP.getKnownSlotsCount() == 0); -} - -TEST_CASE("nomination weight", "[scp]") -{ - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - SIMULATION_CREATE_NODE(3); - SIMULATION_CREATE_NODE(4); - SIMULATION_CREATE_NODE(5); - - SCPQuorumSet qSet; - qSet.threshold = 3; - qSet.validators.push_back(v0NodeID); - qSet.validators.push_back(v1NodeID); - qSet.validators.push_back(v2NodeID); - qSet.validators.push_back(v3NodeID); - - TestNominationSCP const nomSCP(v0NodeID, qSet); - - uint64 result = nomSCP.getNodeWeight(v2NodeID, qSet, false); - - REQUIRE(isNear(result, .75)); - - result = nomSCP.getNodeWeight(v4NodeID, qSet, false); - REQUIRE(result == 0); - - SCPQuorumSet iQSet; - iQSet.threshold = 1; - iQSet.validators.push_back(v4NodeID); - iQSet.validators.push_back(v5NodeID); - qSet.innerSets.push_back(iQSet); - - result = nomSCP.getNodeWeight(v4NodeID, qSet, false); - - REQUIRE(isNear(result, .6 * .5)); -} - -class NominationTestHandler : public NominationProtocol -{ - public: - NominationTestHandler(Slot& s) : NominationProtocol(s) - { - } - - void - setPreviousValue(Value const& v) - { - mPreviousValue = v; - } - - void - setRoundNumber(int32 n) - { - mRoundNumber = n; - } - - void - updateRoundLeaders() - { - NominationProtocol::updateRoundLeaders(); - } - - std::set& - getRoundLeaders() - { - return mRoundLeaders; - } - - uint64 - getNodePriority(NodeID const& nodeID, SCPQuorumSet const& qset) - { - return NominationProtocol::getNodePriority(nodeID, qset); - } -}; - -// A test SCPDriver that allows specification of nodes with 0 weight. -class ZeroWeightTestNominationSCP : public TestNominationSCP -{ - public: - std::set mZeroWeightNodes; - - ZeroWeightTestNominationSCP(NodeID const& nodeID, - SCPQuorumSet const& qSetLocal, - std::set const& zeroWeightNodes) - : TestNominationSCP(nodeID, qSetLocal) - , mZeroWeightNodes(zeroWeightNodes) - { - } - - uint64 - getNodeWeight(NodeID const& nodeID, SCPQuorumSet const& qset, - bool isLocalNode) const override - { - if (mZeroWeightNodes.count(nodeID)) - { - return 0; - } - return TestNominationSCP::getNodeWeight(nodeID, qset, isLocalNode); - } -}; - -static SCPQuorumSet -makeQSet(std::vector const& nodeIDs, int threshold, int total, - int offset) -{ - SCPQuorumSet qSet; - qSet.threshold = threshold; - for (int i = 0; i < total; i++) - { - qSet.validators.push_back(nodeIDs[i + offset]); - } - return qSet; -} - -TEST_CASE("updateRoundLeaders handles zero weight nodes", "[scp]") -{ - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - - // 3 nodes total: v0 (local), v1 (normal weight), v2 (zero weight). - SCPQuorumSet qSet; - qSet.threshold = 2; - qSet.validators.push_back(v0NodeID); - qSet.validators.push_back(v1NodeID); - qSet.validators.push_back(v2NodeID); - - auto runScenario = [&](std::set const& zeroWeightNodes) { - ZeroWeightTestNominationSCP nomSCP(v0NodeID, qSet, zeroWeightNodes); - - Slot slot(0, nomSCP.mSCP); - NominationTestHandler nom(slot); - - Value v; - v.emplace_back(uint8_t(42)); - nom.setPreviousValue(v); - - // Ensure that even with many more rounds than validators, - // `updateRoundLeaders` always terminates and never picks a zero-weight - // node as leader. - int const maxRounds = 20; - for (int i = 0; i < maxRounds; i++) - { - nom.setRoundNumber(i); - nom.updateRoundLeaders(); - } - - return nom.getRoundLeaders(); - }; - - SECTION("non-local zero-weight validator is excluded from round leaders") - { - // v2 simulates a LOW-quality validator with zero weight, mimicking - // HerderSCPDriver::getNodeWeight behavior for LOW-quality nodes. - auto const& leaders = runScenario({v2NodeID}); - REQUIRE(leaders.count(v0NodeID) == 1); - REQUIRE(leaders.count(v1NodeID) == 1); - REQUIRE(leaders.count(v2NodeID) == 0); - REQUIRE(leaders.size() == 2); - } - - SECTION("local zero-weight validator is excluded from round leaders") - { - auto const& leaders = runScenario({v0NodeID}); - REQUIRE(leaders.count(v0NodeID) == 0); - REQUIRE(leaders.count(v1NodeID) == 1); - REQUIRE(leaders.count(v2NodeID) == 1); - REQUIRE(leaders.size() == 2); - } -} - -// this test case display statistical information on the priority function used -// by nomination -TEST_CASE("nomination weight stats", "[scp][!hide]") -{ - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - - SIMULATION_CREATE_NODE(3); - SIMULATION_CREATE_NODE(4); - SIMULATION_CREATE_NODE(5); - SIMULATION_CREATE_NODE(6); - - std::vector nodeIDs = {v0NodeID, v1NodeID, v2NodeID, v3NodeID, - v4NodeID, v5NodeID, v6NodeID}; - - int const totalSlots = 1000; - int const maxRoundPerSlot = 5; // 5 -> 15 seconds - int const totalRounds = totalSlots * maxRoundPerSlot; - - auto runTests = [&](SCPQuorumSet qSet) { - std::map wins; - - TestNominationSCP nomSCP(v0NodeID, qSet); - for (int s = 0; s < totalSlots; s++) - { - Slot slot(s, nomSCP.mSCP); - - NominationTestHandler nom(slot); - - Value v; - v.emplace_back(uint8_t(s)); // anything will do as a value - - nom.setPreviousValue(v); - - for (int i = 0; i < maxRoundPerSlot; i++) - { - nom.setRoundNumber(i); - nom.updateRoundLeaders(); - auto& l = nom.getRoundLeaders(); - REQUIRE(!l.empty()); - for (auto& w : l) - { - wins[w]++; - } - } - } - return wins; - }; - - SECTION("flat quorum") - { - auto flatTest = [&](int threshold, int total) { - auto qSet = makeQSet(nodeIDs, threshold, total, 0); - - auto wins = runTests(qSet); - - for (auto& w : wins) - { - double stats = double(w.second * 100) / double(totalRounds); - CLOG_INFO(SCP, "Got {}{}", stats, - ((v0NodeID == w.first) ? " LOCAL" : "")); - } - }; - - SECTION("3 out of 5") - { - flatTest(3, 5); - } - SECTION("2 out of 3") - { - flatTest(2, 3); - } - } - SECTION("hierarchy") - { - auto qSet = makeQSet(nodeIDs, 3, 4, 0); - - auto qSetInner = makeQSet(nodeIDs, 2, 3, 4); - qSet.innerSets.emplace_back(qSetInner); - - auto wins = runTests(qSet); - - for (auto& w : wins) - { - double stats = double(w.second * 100) / double(totalRounds); - bool outer = - std::any_of(qSet.validators.begin(), qSet.validators.end(), - [&](auto const& k) { return k == w.first; }); - CLOG_INFO(SCP, "Got {} {}", stats, - ((v0NodeID == w.first) ? "LOCAL" - : (outer ? "OUTER" : "INNER"))); - } - } -} - -TEST_CASE("nomination two nodes win stats", "[scp][!hide]") -{ - int const nbRoundsForStats = 9; - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - - SIMULATION_CREATE_NODE(3); - SIMULATION_CREATE_NODE(4); - SIMULATION_CREATE_NODE(5); - SIMULATION_CREATE_NODE(6); - - std::vector nodeIDs = {v0NodeID, v1NodeID, v2NodeID, v3NodeID, - v4NodeID, v5NodeID, v6NodeID}; - - int const totalIter = 10000; - - // maxRounds is the number of rounds to evaluate in a row - // the iteration is considered successful if validators could - // agree on what to nominate before maxRounds is reached - auto nominationLeaders = [&](int maxRounds, SCPQuorumSet qSetNode0, - SCPQuorumSet qSetNode1) { - TestNominationSCP nomSCP0(v0NodeID, qSetNode0); - TestNominationSCP nomSCP1(v1NodeID, qSetNode1); - - int tot = 0; - for (int g = 0; g < totalIter; g++) - { - Slot slot0(0, nomSCP0.mSCP); - NominationTestHandler nom0(slot0); - - Slot slot1(0, nomSCP1.mSCP); - NominationTestHandler nom1(slot1); - - Value v; - v.emplace_back(uint8_t(g)); - nom0.setPreviousValue(v); - nom1.setPreviousValue(v); - - bool res = true; - - bool v0Voted = false; - bool v1Voted = false; - - int r = 0; - do - { - nom0.setRoundNumber(r); - nom1.setRoundNumber(r); - nom0.updateRoundLeaders(); - nom1.updateRoundLeaders(); - - auto& l0 = nom0.getRoundLeaders(); - REQUIRE(!l0.empty()); - auto& l1 = nom1.getRoundLeaders(); - REQUIRE(!l1.empty()); - - auto updateVoted = [&](auto const& id, auto const& leaders, - bool& voted) { - if (!voted) - { - voted = std::find(leaders.begin(), leaders.end(), id) != - leaders.end(); - } - }; - - // checks if id voted (any past round, including this one) - // AND id is a leader this round - auto findNode = [](auto const& id, bool idVoted, - auto const& otherLeaders) { - bool r = (idVoted && std::find(otherLeaders.begin(), - otherLeaders.end(), - id) != otherLeaders.end()); - return r; - }; - - updateVoted(v0NodeID, l0, v0Voted); - updateVoted(v1NodeID, l1, v1Voted); - - // either both vote for v0 or both vote for v1 - res = findNode(v0NodeID, v0Voted, l1); - res = res || findNode(v1NodeID, v1Voted, l0); - } while (!res && ++r < maxRounds); - - tot += res ? 1 : 0; - } - return tot; - }; - - SECTION("flat quorum") - { - // test using the same quorum on all nodes - auto flatTest = [&](int threshold, int total) { - auto qSet = makeQSet(nodeIDs, threshold, total, 0); - - for (int maxRounds = 1; maxRounds <= nbRoundsForStats; maxRounds++) - { - int tot = nominationLeaders(maxRounds, qSet, qSet); - double stats = double(tot * 100) / double(totalIter); - CLOG_INFO(SCP, "Win rate for {} : {}", maxRounds, stats); - } - }; - - SECTION("3 out of 5") - { - flatTest(3, 5); - } - SECTION("2 out of 3") - { - flatTest(2, 3); - } - } - - SECTION("hierarchy") - { - SECTION("same qSet") - { - auto qSet = makeQSet(nodeIDs, 3, 4, 0); - - auto qSetInner = makeQSet(nodeIDs, 2, 3, 4); - qSet.innerSets.emplace_back(qSetInner); - - for (int maxRounds = 1; maxRounds <= nbRoundsForStats; maxRounds++) - { - int tot = nominationLeaders(maxRounds, qSet, qSet); - double stats = double(tot * 100) / double(totalIter); - CLOG_INFO(SCP, "Win rate for {} : {}", maxRounds, stats); - } - } - SECTION("v0 is inner node for v1") - { - auto qSet0 = makeQSet(nodeIDs, 3, 4, 0); - auto qSetInner0 = makeQSet(nodeIDs, 2, 3, 4); - qSet0.innerSets.emplace_back(qSetInner0); - - // v1's qset: we move v0 into the inner set - auto qSet1 = qSet0; - REQUIRE(qSet1.validators[0] == v0NodeID); - std::swap(qSet1.validators[0], qSet1.innerSets[0].validators[0]); - - for (int maxRounds = 1; maxRounds <= nbRoundsForStats; maxRounds++) - { - int tot = nominationLeaders(maxRounds, qSet0, qSet1); - double stats = double(tot * 100) / double(totalIter); - CLOG_INFO(SCP, "Win rate for {} : {}", maxRounds, stats); - } - } - } -} -} diff --git a/src/simulation/ApplyLoad.cpp b/src/simulation/ApplyLoad.cpp index 720f030d86..f96126f9b5 100644 --- a/src/simulation/ApplyLoad.cpp +++ b/src/simulation/ApplyLoad.cpp @@ -1050,16 +1050,13 @@ ApplyLoad::closeBenchmarkLedger(std::vector const& txs, xdr::xdr_from_opaque(wireBytes, receivedXdr); auto receivedTxSet = TxSetXDRFrame::makeFromWire(receivedXdr); - // Nomination through externalization and apply use production SCP paths. + // Proposal through externalization and apply use production SCP paths. herder.getPendingEnvelopes().putTxSet(receivedTxSet->getContentsHash(), ledgerSeq, receivedTxSet); auto& driver = herder.getHerderSCPDriver(); - driver.nominate( - ledgerSeq, - [&driver, nominatedValue]() { - return driver.wrapStellarValue(nominatedValue); - }, - lcl.header.scpValue); + driver.startSlot(ledgerSeq, [&driver, nominatedValue]() { + return driver.wrapStellarValue(nominatedValue); + }); auto const deadline = std::chrono::steady_clock::now() + std::chrono::seconds(60); diff --git a/src/test/TxTests.cpp b/src/test/TxTests.cpp index 66b40cc8c4..d4e4a68c12 100644 --- a/src/test/TxTests.cpp +++ b/src/test/TxTests.cpp @@ -80,7 +80,7 @@ captureLastClosedLedgerLcm(Application& app) // Configs whose ledger content depends on thread scheduling or on // randomized nomination cannot produce reproducible golden data. if (app.getConfig().EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD || - app.getConfig().TESTING_NOMINATE_RANDOM_VALUES) + app.getConfig().TESTING_PROPOSE_RANDOM_VALUES) { disableLcmCapture(); } From b524ef9624f9212153adb290aec20542cac94119 Mon Sep 17 00:00:00 2001 From: Garand Tyson Date: Tue, 10 Nov 2026 20:51:59 +0000 Subject: [PATCH 2/2] Describe the leader-schedule prototype and its perfnet results in the README Replace the stock README on this experimental branch with a description of the change, the 2 s (pass) and 1 s (fail) perfnet results measured on daa965f4c, and how they were run: topology, hardware, node configuration, network settings, load and pass criteria. --- README.md | 262 +++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 238 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index 0126b7350e..f6d812cfd7 100644 --- a/README.md +++ b/README.md @@ -1,34 +1,248 @@ -
-Stellar -
-Creating equitable access to the global financial system -

Stellar Core

-
-

-Build Status -

+# stellar-core: leader schedule without SCP nomination (experimental branch) -Stellar-core is a replicated state machine that maintains a local copy of a cryptographic ledger and processes transactions against it, in consensus with a set of peers. -It implements the [Stellar Consensus Protocol](https://github.com/stellar/stellar-core/blob/master/src/scp/README.md), a _federated_ consensus protocol. -It is written in C++20 and runs on Linux, OSX and Windows. -Learn more by reading the [overview document](https://github.com/stellar/stellar-core/blob/master/docs/README.md). +**This branch is an experiment. It is not production code and must not run on a real network.** +It has no feature flag, no protocol gate, and it is not compatible with nodes that nominate. -# Documentation +Results below were measured on commit `daa965f4c` on 2026-11-10 and are one run per setting. -Documentation of the code's layout and abstractions, as well as for the -functionality available, can be found in -[`./docs`](https://github.com/stellar/stellar-core/tree/master/docs). +## What this branch is -# Installation +It is the overlay-v2 stack up to [#5459](https://github.com/stellar/stellar-core/pull/5459) +(Rust overlay over QUIC, millisecond close times, compressed tx-set responses, leader-only tx-set +construction) plus one change: **the SCP nomination phase is deleted and replaced by a leader +schedule that every node computes locally.** Draft PR: +[#5495](https://github.com/stellar/stellar-core/pull/5495). -See [Installation](./INSTALL.md) +- **Schedule.** For slot N, the nodes of the transitive quorum are sorted by + `SHA256(hash of ledger header N-1 || node id)`. The leader of round r is entry `(r-1) mod n`. +- **Rounds.** A round is a ballot counter. The round's leader builds its value and goes straight to + the ballot protocol with `(r, value)`. The other nodes vote for the leader's ballot once it passes + proposal validation. A timeout (the ballot timeout settings) hands the slot to the next leader. +- **Unchanged.** The ballot protocol's accept, confirm and commit rules, value validation, apply, + ledger state and meta. -# Contributing +The question it answers: how much of a ledger's time is nomination, and what limits block time once +nomination is gone? -See [Contributing](./CONTRIBUTING.md) +## Performance results -# Running tests +| Target block time | Load | Verdict | Block time measured (average over 30 nodes) | +|---|---|---|---| +| **2,000 ms** | 3,000 TPS | **PASS**, all four windows | p75 2,066-2,109 ms, p99 2,286-2,324 ms | +| **1,000 ms** | 3,000 TPS | **FAIL**, first window | p75 1,386 ms (limit 1,200), p99 1,599 ms | -See [running tests](./CONTRIBUTING.md#running-tests) +In short: at 2 s the network keeps up and has more slack per ledger than with nomination. At 1 s it +settles at about 1.33 s per ledger. Nothing breaks; the stages of one ledger simply add up to more +than a second. -[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/stellar/stellar-core) +### 2,000 ms target: PASS + +Load ran its full 900 s. Every node passed every window. + +| Window | p75, average (range over nodes) | p99, average (range over nodes) | +|---|---|---| +| 1 | 2,109 ms (2,086-2,128) | 2,324 ms (2,282-2,359) | +| 2 | 2,066 ms (2,058-2,076) | 2,286 ms (2,239-2,328) | +| 3 | 2,082 ms (2,067-2,091) | 2,296 ms (2,251-2,333) | +| 4 | 2,094 ms (2,080-2,105) | 2,303 ms (2,262-2,344) | + +Checks, all passed: +- **Inclusion:** every node's ledgers hold exactly the 2,880,000 offered transactions. +- **No fork**, all 30 nodes in sync, still closing ledgers after the load. +- **Error counters zero** (invalid signatures, invariant failures, publish failures, invalid values, + lost sync), no node restarts. +- **No wasted rounds:** every ledger was decided by the first leader. 0 leader timeouts, 0 ballot + timeouts, 0 empty transaction sets. + +Against the same fixture on #5459's head `ebdd73635` (with nomination), run the same day on the same +instance type: + +| 5-minute means, median node | #5459 `ebdd73635` | This branch `daa965f4c` | +|---|---:|---:| +| Verdict | PASS | PASS | +| p75 per window | 2,112 / 2,099 / 2,101 / 2,099 ms | 2,109 / 2,066 / 2,082 / 2,094 ms | +| p99 per window | 2,334 / 2,391 / 2,324 / 2,354 ms | 2,324 / 2,286 / 2,296 / 2,303 ms | +| Ledgers per 5 minutes | 147-148 | 149-150 | +| Transactions per ledger | 6,092 | 6,003 | +| Trigger to first ballot | 1.21 s | 0.88 s | +| First ballot to externalize | 0.29 s | 0.43 s | +| **Trigger to externalize** | **1.50 s** | **1.31 s** | +| Leader tx-set build | 0.28 s | 0.27 s | +| Tx-set fetch (follower) | 0.24 s | 0.26 s | +| Tx-set validation (follower) | 0.20 s | 0.19 s | +| Ledger close | 0.164 s | 0.162 s | + +- Block time is the same on both, as expected: when the network keeps up, the 2 s trigger sets the + cadence. +- The time from trigger to externalize is 0.19 s shorter. The part before the first ballot that is + not build, fetch or validation fell from about 0.48 s (nomination rounds) to about 0.15 s. +- Some of that came back in the ballot phase. Nodes now start their ballot at different times, as + each finishes validating the leader's set, so the early ones wait for a quorum of the others. + +### 1,000 ms target: FAIL + +The first 5-minute window missed on all 30 nodes, and the test stops the load at the first miss. + +| Window 1 | Value | +|---|---| +| p75, average (range over nodes) | 1,386 ms (1,375-1,399); the limit is below 1,200 ms | +| p99, average (range over nodes) | 1,599 ms (1,582-1,628); the limit is 2,000 ms | +| Ledgers per 5 minutes | 225 (1.33 s each) | +| Transactions per ledger | 3,750, the cap, on every ledger | +| Leader timeouts, ballot timeouts, empty tx sets | 0, 0, 0 | +| Error counters, in sync, no fork, still closing | all passed | + +The network was over capacity at this block time: 3,000 TPS x 1.33 s is 4,000 transactions offered +per ledger against a cap of 3,750, so the mempool backed up and every ledger started the instant +the previous one closed. Inclusion is not checked after an early stop. No baseline was run at this +setting. + +### Where a ledger's time goes + +| Stage (5-minute means, median node) | 2,000 ms target (6,000 txs) | 1,000 ms target (3,750 txs) | +|---|---:|---:| +| Leader builds the tx set | 0.27 s | 0.25 s | +| Trigger to first ballot (includes the build, the fetch and validation) | 0.88 s | 0.70 s | +| First ballot to externalize | 0.43 s | 0.41 s | +| Ledger close (150 ms / 187 ms of it is simulated apply) | 0.16 s | 0.20 s | +| **Sum** | **1.47 s** | **1.30 s** | + +Halving the transaction set took only 0.17 s off the cycle. What remains is mostly cross-region +round trips, not CPU: + +1. **Tx-set dissemination takes about 2.5 round trips.** A follower receives the proposal, requests + the set from the proposer, and the response takes two flights. Seen from us-west-2, the fetch + took 27 ms from the same region, 150 ms from us-east-1, 301 ms from eu-central-1 and 362 ms from + ap-southeast-1. Nothing is pushed, and transactions are sent in full although followers already + hold nearly all of them. +2. **Followers validate before they vote, and the ballot then needs four one-way trips.** +3. **About 70 ms pass between the leader finishing its set and the proposal reaching a peer in the + same region.** SCP state, including the tx set, is written to the database before each + broadcast. That this write accounts for most of the 70 ms is inferred, not timed. +4. **The leader's build is on the critical path.** At 1 s nothing can be built ahead of the trigger. + +## How it was measured + +Both runs use the `block_time_check` mission of perfnet, SDF's tooling for performance networks on +AWS. Each run launches a fresh network, brings it to the target settings, runs the load, judges +it, and terminates the network. + +### Network topology + +| | | +|---|---| +| Validators | 30, all validating, no watchers | +| Organizations | 10 home domains of 3 validators each, all `HIGH` quality | +| Quorum set | Generated by stellar-core from the home domains: 7 of 10 organizations, 2 of 3 validators within each | +| Regions | 4 AWS regions: eu-central-1 (8 nodes), ap-southeast-1 (8), us-east-1 (7), us-west-2 (7) | +| Placement | 9 organizations have their three validators in three different regions; one has two in us-east-1 and one in ap-southeast-1. The layout was identical in both runs | +| Connectivity | Full mesh: every node is configured with the other 29 as its only peers | +| Round-trip times seen from us-west-2 | us-east-1 about 60 ms, eu-central-1 about 147 ms, ap-southeast-1 about 180 ms | + +### Hardware and software + +| | | +|---|---| +| Instance type | c6id.2xlarge: 8 vCPU (Intel Ice Lake, with SHA extensions), 16 GiB RAM, local NVMe | +| Build | Commit `daa965f4c`, the perftests package `30.0.1-3674.daa965f4ce.noble~perf.leader.schedule.no.nomination~perftests`. All 30 nodes reported that commit in their build label | +| Overlay | The Rust overlay of this stack: QUIC, one connection per peer | +| Ledger protocol | 30, upgraded from genesis during preparation | +| Database | SQLite | + +### Node configuration + +Beyond identity, peers and home domains, every node ran with: + +``` +ARTIFICIALLY_GENERATE_LOAD_FOR_TESTING=true +UNSAFE_QUORUM=true +FAILURE_SAFETY=0 +GENESIS_TEST_ACCOUNT_COUNT=1000000 +TARGET_PEER_CONNECTIONS=29 +MAX_ADDITIONAL_PEER_CONNECTIONS=87 +PREFERRED_PEERS_ONLY=true +FLOOD_DEMAND_BACKOFF_DELAY_MS=1000 +INVARIANT_CHECKS=[] +TESTING_IGNORE_LEDGER_TIME_UPGRADE_BOUNDS=true +TESTING_MAX_CLASSIC_BYTE_ALLOWANCE=1048576 +TESTING_MAX_SOROBAN_BYTE_ALLOWANCE=9437184 +LOADGEN_PREGENERATED_TRANSACTIONS_FILE="/var/lib/stellar-core/loadgen-txs.xdr" +OP_APPLY_SLEEP_TIME_DURATION_FOR_TESTING=[25] # [50] in the 1,000 ms run +OP_APPLY_SLEEP_TIME_WEIGHT_FOR_TESTING=[100] +``` + +`EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD` was off (the default). Catchup and automatic restart are +off, so a node that crashes or falls behind stays that way and fails the run. + +### Network settings (applied by upgrade before the load) + +| Setting | 2,000 ms run | 1,000 ms run | +|---|---:|---:| +| Target ledger close time | 2,000 ms | 1,000 ms | +| SCP timeouts (ballot and nomination, initial and increment) | 2,000 ms | 2,000 ms | +| Maximum transactions per ledger (125% of the offered load per block) | 7,500 | 3,750 | +| Maximum tx-set size in bytes | 10 MiB (1 MiB classic + 9 MiB Soroban) | same | +| Soroban limits | raised to fit the load | raised to fit the load | + +The leader timer of this branch uses the ballot timeout, so a round's leader had 2 s (4 s with its +one extension) in both runs. + +### Load + +| | | +|---|---| +| Transactions | Stellar Asset Contract (SAC) payments, pregenerated; no classic transactions | +| Rate | 3,000 TPS offered: every validator generates 100 TPS | +| Accounts | 1,000,000 genesis accounts, 33,333 per generator | +| Duration | 60 s warm-up, then 900 s measured | +| Apply | Skipped ("overlay-only" mode) and replaced by a sleep per operation, sized as a 150 ms budget per ledger at the offered load: 25 us per operation at 2,000 ms, 50 us at 1,000 ms | + +Commands (the two runs differ only in the block time): + +``` +block_time_check.py \ + --stellar-core-version 30.0.1-3674.daa965f4ce.noble~perf.leader.schedule.no.nomination~perftests \ + --block-time-ms 2000 \ # 1000 for the second run + --soroban-tps 3000 --soroban-type sac --classic-tps 0 \ + --simulate-apply-budget-ms 150 --load-duration-s 900 \ + --instance-type c6id.2xlarge --save-logs always +``` + +Defaults that matter: 10 organizations, validators as generators, `--scp-timeout-ms 2000`, +`--max-tx-set-bytes 10485760`, 1,000,000 genesis accounts, seed 0. + +### How a run is judged + +- Ledger close times are read per node from Prometheus (scraped every 10 s) and judged over + 5-minute windows after the warm-up. +- **PASS** needs, on every node in every window: p75 within [0.8 T, 1.2 T), p99 at most 2 T, and at + least one close per 2 T on average, where T is the target block time. It also needs every node's + ledgers to hold exactly the offered transactions, zero error counters, all nodes in sync, no + fork, and the network still closing ledgers afterwards. +- The load stops at the first window that misses, so a FAIL can rest on one window. + +### Limits of these numbers + +- One run per setting, and one baseline run (at 2,000 ms only). +- Apply is simulated, so these are consensus and overlay numbers, not end-to-end ledger close. +- `scp.timing.trigger-to-ballot` records 0 when a follower votes before its own trigger fires, so it + is not strictly comparable with the nomination build. Triggers were on time in the 2,000 ms run. +- The per-stage timeline and the fetch times by region come from one node's logs at 2,000 ms and + four nodes' logs at 1,000 ms. + +## Known limitations of the prototype + +- The leader schedule's node set is whatever transitive quorum each node currently knows. It + assumes a symmetric, static quorum. +- Upgrades: a value needs a quorum of nodes that armed the same upgrades; there is no stripping or + merging. A follower follows a leader's empty-tx-set ballot under the weaker ballot validation. +- Leader grinding through the header hash and fairness between organizations are out of scope. +- The leader timer extension is broader than intended, and a proposal that arrives just after a + timeout is wasted. Neither occurred in these runs. +- A build with `--disable-tests` does not compile at this commit (test headers reached from + `herder/HerderImpl.cpp`). + +## Building and the rest of the documentation + +Unchanged from stellar-core: see [INSTALL.md](./INSTALL.md), [CONTRIBUTING.md](./CONTRIBUTING.md) +and [`./docs`](./docs). The full test suite passes on `daa965f4c`, also with `--all-versions`.