Skip to content

Prototype: replace SCP nomination with a deterministic leader schedule - #5495

Merged
SirTyson merged 2 commits into
stellar:pr/merge-ms-close-time-basefrom
SirTyson:perf/leader-schedule-no-nomination
Nov 10, 2026
Merged

SirTyson merged 2 commits into
stellar:pr/merge-ms-close-time-basefrom
SirTyson:perf/leader-schedule-no-nomination

Conversation

@SirTyson

@SirTyson SirTyson commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Prototype, not for merge. This deletes the SCP nomination phase and replaces it with a leader schedule that every node computes locally. There is no flag, no protocol gate and no compatibility with nominating nodes: it exists to measure what nomination costs on the overlay-v2 stack and what limits block time once it is gone.

Stacked on #5459 (pr/txset-compression, ebdd73635). The change is one commit, daa965f4c. A second commit, 577f3d4f0, only replaces the branch's README.md with a description of the experiment, the results below and the full test setup (topology, node configuration, network settings, load, pass criteria); everything here was built, tested and measured on daa965f4c.

Result in one paragraph

On perfnet (30 validators, 4 AWS regions, c6id.2xlarge, 3,000 SAC TPS) the prototype passes at a 2 s target with every correctness check green and no wasted round, and takes about 0.19 s out of the time from trigger to externalize (1.50 s to 1.31 s). Block time at 2 s is unchanged, because the trigger sets the cadence when the network keeps up. At a 1 s target it fails: the network settles at 1.33 s per ledger. The remaining time is mostly cross-region round trips for the tx set and the ballot, not nomination and not CPU. One run per setting.

Design

  • Schedule. For slot N, sort the transitive quorum by SHA256(LCL(N-1).hash || nodeID); leader(N, r) is entry (r-1) mod n. Uniform, no weights, one sort per slot (herder/LeaderSchedule.{h,cpp}, cached in HerderSCPDriver).
  • Rounds. A round is a ballot counter. The round's leader builds its value and bumps straight to (r, value); the others adopt the leader's PREPARE/CONFIRM at counter r once it passes proposal validation. A timeout (the ballot timeout settings) moves a node to round r+1.
  • Late proposals. A node that has not voted yet still votes for an earlier round leader's ballot, at that ballot's own counter, when it shows up late (up to 8 rounds back). Nomination had this property; without it a slow first proposal is wasted and the next leader builds a competing set.
  • Leader timer extension. If the timer fires while the Herder already holds the leader's statement (its tx set is being fetched or it is queued), the timer is re-armed once.
  • Unchanged: the ballot protocol's accept, confirm and commit rules, mValueOverride precedence, the v-blocking join paths, value validation, apply, ledger state and meta.
  • NOMINATE envelopes are dropped.

Why it should be safe (and where it is not a real design)

  • Safety rests on the ballot protocol, which is safe for any choice of ballot values. The change only alters which value a node picks for a new ballot and when it bumps. Two independent reviews found no path to two honest nodes externalizing different values, no vote the ballot protocol forbids, and no new reachable assert or unbounded loop.
  • Upgrade governance: adoption validates in the strict (former nomination) mode, so a value needs a quorum of adopters that armed the same upgrades. There is no upgrade stripping or merging.
  • Known gaps, by design of a prototype:
    • A follower follows a leader's empty-tx-set ballot under ballot validation, so its upgrades are only checked for apply validity. With parallel tx-set download on, one round leader could get an empty ledger adopted network-wide.
    • The schedule's node set is whatever transitive quorum each node currently knows. It assumes a symmetric, static quorum (as on perfnet). A real design needs a node set that is part of consensus state.
    • Leader grinding through the header hash, and fairness by org or quality, are out of scope.

What was deleted

  • scp/NominationProtocol.{h,cpp}, scp/test/SCPUnitTests.cpp, and the nomination API of SCP, Slot and SCPDriver (nominate, combineCandidates, computeHashNode, getNodeWeight, the nomination timers and emit delay, and related helpers).
  • HerderSCPDriver::combineCandidates / compareTxSets / getNodeWeight, the metrics scp.timing.nominated, scp.timeout.nominate, scp.nomination.combinecandidates, and ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING.
  • Kept but unused, so existing configs still parse: FORCE_OLD_STYLE_LEADER_ELECTION, VALIDATOR_WEIGHT_CONFIG, nominationtimeoutlimit and the nomination timeout network settings.
  • New metrics: scp.timeout.leader (leader timeouts per slot; an extension counts) and scp.leader.round (the highest round a node entered for the slot, so an upper bound on the round that externalized).

Net: 23 non-test files, +886 / -2,072.

Testing (all on daa965f4c)

  • New: [scp][leaderschedule] unit cases and a randomized multi-node model in scp/test/SCPTests.cpp, herder/test/LeaderScheduleTests.cpp, and network simulations in herder/test/LeaderRoundsTests.cpp (including failover when the dead node leads round 1). The Herder, quorum-tracker, PendingEnvelopes and overlay-IPC tests are ported; nomination statistics, combineCandidates and weight tests are deleted with the code.
  • Full suite: PASS. 598 test cases / 2,379,273 assertions, libsodium 77, xdrpp 7, check-rust and the soroban host tests (103 test result: ok, 0 failed).
  • Full suite --all-versions: PASS. 598 test cases / 8,154,187 assertions. (Compress and share transaction set responses #5459's head runs 605; the difference is the deleted nomination tests.)
  • Tag run [herder],[scp],[txset],[overlay-ipc],[overlay],[ipc],[upgrades],[simulation] (146 tests): passes except two [acceptance] tests, "hierarchical topology scales 1 to 3" and "core nodes with outer nodes", which time out the same way on overlay-v2-shared without this change and are not in the full suite.
  • clang-format 20.1.8: the 28 added or modified C++ files are clean.
  • Not passing: a --disable-tests build stops in herder/HerderImpl.cpp, on test headers reached through #include "simulation/LoadGenerator.h" (line 37). That include is the same on Compress and share transaction set responses #5459's head and this commit does not touch those headers; the base was not built with that flag to confirm, and because the build stops there it is not known whether this commit's own files compile without tests.
  • Not run: sanitizers; the tx-meta and LedgerCloseMeta determinism checks (the change does not touch apply).
  • Test gaps (from review): no SCP-level test for following an empty-tx-set leader ballot, for the UINT32_MAX counter case, for the mValueOverride restore, for the extension flag resetting across rounds, or for the 8-round bound. The randomized model uses one shared schedule and never exercises the pending-proposal path. Two hidden trigger-timer tests that depended on the nomination emit delay were deleted without a port.

Perfnet

30 validators in 4 regions, 10 tier-1 orgs, c6id.2xlarge, SAC payments, 150 ms simulated apply per ledger, 900 s measured, all SCP timeouts 2 s (perfnet's default). Baseline is #5459's head ebdd73635 on the same fixture and instance type, run the same day. One run per setting.

2 s target, 3,000 TPS: PASS

All four windows on target; inclusion (2,880,000 txs on every node), error counters, in sync, no fork and still closing all passed.

#5459 ebdd73635 This PR daa965f4c
avg p75 per window 2,112 / 2,099 / 2,101 / 2,099 ms 2,109 / 2,066 / 2,082 / 2,094 ms
avg p99 per window 2,334 / 2,391 / 2,324 / 2,354 ms 2,324 / 2,286 / 2,296 / 2,303 ms
Ledgers per 5 min 147-148 149-150
Trigger to first ballot (mean) 1.21 s 0.88 s
Ballot to externalize (mean) 0.29 s 0.43 s
Trigger to externalize 1.50 s 1.31 s
Leader build / tx-set fetch / validation (mean) 0.28 / 0.24 / 0.20 s 0.27 / 0.26 / 0.19 s
Ledger close 0.164 s 0.162 s
Nomination timeouts (network, one window) 7 n/a
Leader timeouts, prepare timeouts, rounds above 1, empty tx sets n/a 0, 0, none, 0
  • The time before the first ballot that is not build, fetch or validation fell from about 0.48 s to about 0.15 s.
  • Part of it came back in the ballot phase: nodes now start their ballot when each has validated the leader's set, so the early ones wait for a quorum of the others.
  • scp.timing.trigger-to-ballot records 0 when a follower votes before its own trigger, so it is not strictly comparable. Triggers were on time in this run.

1 s target, 3,000 TPS: FAIL

Window 1 missed on all 30 nodes: avg p75 1,386 ms (limit 1,200), avg p99 1,599 ms. 225 ledgers per 5 minutes (1.33 s each), every tx set at the 3,750 cap, every trigger already due, mempool backed up. No leader or prepare timeouts, every ledger decided in round 1, no empty tx sets; error counters, in sync, no fork and still closing passed. No baseline was run at this setting.

Stage (5-min means) 2 s target (6,000 txs) 1 s target (3,750 txs)
Leader build 0.27 s 0.25 s
Trigger to first ballot 0.88 s 0.70 s
Ballot to externalize 0.43 s 0.41 s
Ledger close (simulated apply 150 / 187 ms) 0.16 s 0.20 s
Sum 1.47 s 1.30 s

Where the time goes now

From one node's logs at the 2 s setting (456 ledgers as follower, 23 as leader) and live logs from four regions at 1 s:

  1. Tx-set dissemination is 2.5 round trips. A follower receives the proposal, requests the set from the proposer, and the response (0.77 MB zstd for 6,000 txs) takes two flights: fetch measured 27 ms in-region, 150 ms from us-east-1, 301 ms from eu-central-1 and 362 ms from ap-southeast-1, as seen from us-west-2. Nothing is pushed, and full transactions are sent although followers hold nearly all of them.
  2. Followers validate before voting, then the ballot needs four one-way trips (0.29 s on the baseline, where all nodes start together).
  3. About 70 ms pass between "Built TX set" and the proposal reaching a same-region peer. emitEnvelope calls persistSCPState before broadcast, which serializes and base64-encodes the tx set and writes it to the database on the main thread. That this write is most of the 70 ms is inferred, not timed.
  4. The leader's build is on the critical path (0.25-0.33 s). The early build is discarded unless the set was capacity limited, and it does not run at all when the trigger is already due.
  5. Validation is 0.12-0.19 s with signatures already cached.

EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD does not help here as it stands: followers would confirm prepared before holding the set, and the ballot protocol does not retry the commit vote at the same counter, so each slot would wait for the ballot timeout.

Open review findings (not fixed in this commit)

  • The leader timer extension triggers on any pending envelope from the round's leader for the slot, not only its proposal for that round. It slows failover when a tx set is unfetchable; it does not occur on a healthy network.
  • A late proposal only helps nodes without a ballot, never the next leader, so a proposal that arrives just after a timeout costs a second fetch and validation.
  • A leader re-proposes a value that is only structurally valid; bumpState may replace it with the empty-tx-set value.
  • Nodes that adopted a ballot but have not heard a quorum run no timer, as in the plain ballot protocol, so recovery after a timeout depends on the next leader being alive.
  • Smaller items: a redundant progressRound() before advanceSlot, repeated proposal validation while a leader has no ballot, leftover diagnostics in one Herder test, stale "nominate" wording in a few comments.

Next steps

Not started. In order of expected effect on block time: push the tx set with the proposal in one flight; take SCP state persistence off the broadcast path (after timing it); let the commit vote resume when a late tx set validates, so the prepare rounds can overlap with fetch and validation; build the leader's set before the previous ledger closes, which needs the leader to be known earlier than the previous header hash. Then repeat the 1 s run, with a same-setting baseline and more than one run per build.

@SirTyson
SirTyson force-pushed the perf/leader-schedule-no-nomination branch from 577f3d4 to 876b10a Compare November 10, 2026 21:05
@SirTyson
SirTyson force-pushed the pr/txset-compression branch from ebdd736 to dfd2e66 Compare November 10, 2026 21:05
SirTyson and others added 2 commits November 10, 2026 13:22
Experimental. Not protocol-gated and not compatible with nominating nodes.

Nomination is removed. For slot N every node orders the transitive quorum
by SHA256(hash of ledger header N-1 || node id); the leader of round r (a
round is a ballot counter, starting at 1) is entry (r - 1) mod n. The round
leader builds a value and proposes it straight into the ballot protocol as
PREPARE at counter r. Other nodes vote for the leader's ballot at that
counter if it passes proposal validation (the former nomination-time
checks, e.g. only upgrades the node armed). A leader timer (while the node
has no ballot for its round) or the ballot timer moves a node to the next
round. Accept, confirm and commit are unchanged, and every new vote still
goes through bumpState (value override, updateCurrentValue).

Details:
- Followers vote for the round 1 leader as soon as they see its ballot,
  without waiting for their own trigger.
- A node that has not voted yet votes for an earlier round's leader ballot
  at that ballot's counter when it arrives late, and keeps waiting for its
  current round's leader.
- The leader timer gets one extension when the Herder already holds the
  leader's statement (e.g. it is fetching its transaction set).
- A leader's empty-tx-set ballot is followed under ballot validation.
- No leader and no leader timer at the UINT32_MAX counter.
- State restore sets the value override and the round.
- Herder: the trigger starts the slot; the early tx-set build and the
  follower signature prewarm key off the round leader; NOMINATE envelopes
  are discarded. New metrics scp.timeout.leader and scp.leader.round;
  scp.timing.nominated, scp.timeout.nominate and
  scp.nomination.combinecandidates are gone.

Removed: NominationProtocol, the nomination API of SCP/Slot/SCPDriver,
combineCandidates, node weights, upgrade stripping, the nomination emit
delay, and the nomination-only tests (SCPUnitTests.cpp, nomination cases
of SCPTests.cpp, weight / win-rate / timeout statistics in HerderTests.cpp).
FORCE_OLD_STYLE_LEADER_ELECTION, VALIDATOR_WEIGHT_CONFIG and the nomination
timeout settings still parse but are unused.

Tests: leader round unit cases and a randomized multi-node model in
SCPTests.cpp, LeaderScheduleTests.cpp, LeaderRoundsTests.cpp (network
simulations, including failover with a validator down); Herder, quorum
tracker, PendingEnvelopes and overlay IPC tests ported to leader rounds.

Known limitations: symmetric, static quorums are assumed (the schedule's
node set is each node's view of the transitive quorum); uniform leader
weights; the header hash can be ground by a leader; no upgrade stripping
or merging; a leader re-proposes a value that is only structurally valid.
… README

Replace the stock README on this experimental branch with a description of
the change, the 2 s (pass) and 1 s (fail) perfnet results measured on
daa965f, and how they were run: topology, hardware, node configuration,
network settings, load and pass criteria.
@SirTyson
SirTyson force-pushed the perf/leader-schedule-no-nomination branch from 876b10a to b524ef9 Compare November 10, 2026 21:22
@SirTyson
SirTyson force-pushed the pr/txset-compression branch from dfd2e66 to 1a52bf6 Compare November 10, 2026 21:22
@SirTyson
SirTyson deleted the branch stellar:pr/txset-compression November 10, 2026 22:04
@SirTyson SirTyson closed this Nov 10, 2026
@SirTyson SirTyson reopened this Nov 10, 2026
@SirTyson
SirTyson changed the base branch from pr/txset-compression to pr/merge-ms-close-time-base November 10, 2026 22:06
@SirTyson
SirTyson changed the base branch from pr/merge-ms-close-time-base to overlay-v2-shared November 10, 2026 22:07
@SirTyson
SirTyson changed the base branch from overlay-v2-shared to pr/merge-ms-close-time-base November 10, 2026 22:09
@SirTyson
SirTyson marked this pull request as ready for review November 10, 2026 22:09
@SirTyson
SirTyson merged commit 2b2c8b0 into stellar:pr/merge-ms-close-time-base Nov 10, 2026
7 checks passed
SirTyson added a commit that referenced this pull request Nov 10, 2026
…der schedule (#5496)

Brings the stacked PRs that were merged into
`pr/merge-ms-close-time-base` into `overlay-v2-shared`:

- #5457 Batch GETDATA transaction responses
- #5458 Select loadgen source accounts in constant time
- #5459 Compress and share transaction set responses
- #5495 Prototype: replace SCP nomination with a deterministic leader
schedule
SirTyson added a commit that referenced this pull request Nov 11, 2026
**Prototype, not for merge as is.** Like #5495 it has no flag and no
protocol gate. It moves tx-set dissemination from a request model to a
push model: the round's leader sends the tx set it is about to propose
to its peers as soon as the set is built, instead of waiting for each
follower to ask for it.

Stacked on #5495. The branch sits on `b524ef962`, the head #5495 had
when it was merged; `overlay-v2-shared` (`cc51aba48`, #5496) has the
same tree, so this PR shows only its own four commits:

- `0f9255d47` Start QUIC connections with an 8 MiB congestion window
- `8f7768be7` Push the leader's tx set to peers when it is built
(prototype)
- `d0fe62652` Document the remaining gaps of the tx-set push (docs and
one `.gitattributes` line, no code)
- `2f9baef24` Describe the tx-set push experiment and its perfnet
results in the README (the branch `README.md` now covers this
experiment, with #5495's values as "before"; README only)

Everything below was built, tested and measured on `8f7768be7`.

## Result in one paragraph

On perfnet (30 validators, 4 AWS regions, c6id.2xlarge, 3,000 SAC TPS)
at the 1 s target that #5495 failed, all four windows are on target on
every node: avg p75 1,091 / 1,096 / 1,109 / 1,115 ms against 1,386 ms,
and 1.05 s per ledger against 1.33 s. A follower's Core gets the
leader's tx set 16 ms after it asks for it, where it took 217 ms. One
caveat: every node's ledgers hold 60 transactions more than the
2,880,000 that were offered. The counters point to about 60 transactions
being included in two ledgers, which only the fixture's overlay-only
mode allows; see "The inclusion check" below. The run was judged with
the exact check and reported FAIL for that check alone. It is accepted
as an artifact of that mode, and the check is being relaxed to allow
such duplicates (stellar-experimental/performance-testing#4). One run,
against one run of the base from the same day.

## Design

- **Trigger.** A new Core-to-overlay IPC message `PUSH_TX_SET
[hash][slot]`. `HerderImpl` sends it right after `CACHE_TX_SET` from
inside the build when the node leads the current round, after the build
for a first-round leader's capacity-limited prepared set (the one case
where the prepared set is proposed unchanged), and again when a prepared
set is taken at the trigger. Later-round leaders go through the same
build hook. `CACHE_TX_SET` alone does not push: it also carries sets
that are never proposed.
- **Sender.** The overlay writes the cached, already compressed set to
the tx-set stream of every connected peer that has not been sent it. One
push in flight per peer; a peer that does not read is not sent further
sets and requests them.
- **Wire.** The pushed frame is the existing tx-set frame, sent without
a request. No new wire message, no claimed hash, no slot.
- **Receiver.** The overlay decompresses, strict-parses and hashes the
frame as it does a response and caches it under the hash it computed.
Core is not told. When an SCP statement names that hash, Core's
`RequestTxSet` is answered from the cache at once.
- **Push against request.** If the proposal reaches a follower before
the push is complete, the follower requests the set as before. The
leader does not answer the first request from a peer for a set that is
still being written to it or was written less than 500 ms ago: the bytes
are already on the same ordered stream. Every other request is answered.
- **Fallback.** The request path is unchanged. A node that is not
connected to the leader, connected later, or was busy with an earlier
push has no cached set and requests it; Core's 1 s retry to another peer
is still behind that. The push is not relayed: a node that is not
connected to the leader asks the peer that relayed the proposal, which
holds the pushed copy.
- **One flight.** quinn paces one congestion window over 0.8 round trips
and does not grow the window while the sender is application limited.
Measured locally (two overlay nodes over a loopback with a netem delay,
150 ms round trip, 640 KB frame, one run per setting): with quinn's
default window the frame arrives 229-253 ms after the send on a warm
connection (75 ms of that is the one-way delay) and 965 ms on a cold
one; with an initial window of 8 MiB it arrives after 94 ms. Without
that, a push would lose the race against the proposal (sent about 70 ms
later) on every cross-region link. Hence the first commit. The window
applies to every stream of a connection; it is a prototype setting.
- **Considered and not done:** a compact wire form (transaction hashes,
rebuilt from the follower's mempool; a follower that lacks one
transaction needs another round trip, and at 3,000 TPS the newest
transactions are still flooding); an announcement before the body (a new
way to stall a follower); relaying (in a full mesh every set would
arrive 29 times without an inventory exchange).

## Why it should be safe

- No change to tx-set validation, the voting rules, the ballot protocol
or apply. The C++ diff adds IPC sends on the main thread and metrics.
- A follower only uses a set whose hash is in a signed SCP statement:
the overlay files a pushed set under the hash of its own content, Core
asks only for hashes named by statements it is processing, accepts a set
only while it is fetching that hash, and re-checks the content hash. A
pushed set that does not match the proposal is never looked up; the
proposed set is requested. This is tested at three levels (below).
- What a leader can do with the push: push nothing or late (followers
request, as before); push a set it does not propose (it sits in a
bounded cache and is never used); push to some peers only (the others
request); push garbage or a truncated frame (dropped at decode; a
stalled frame blocks only that peer's own tx-set stream, as a stalled
response does today); push many or large sets (bounded, next point).
- Unrequested tx-set frames were accepted without limits before this
change. They now have a per-peer token bucket applied before
decompression (burst 8, 4 per second; answers to the node's own requests
do not use it) and limits on cached sets from peers that Core has not
asked for (per peer 4 pushed and 16 requested, 128 MiB in total, largest
holder evicted first).
- Known gaps, stated in `docs/rust-overlay/txset-fetching.md`:
- A follower that drops a pushed set on arrival (its own limits) asks
inside the 500 ms in which the leader skips the request, so it is only
served by Core's 1 s retry. Honest leaders do not trigger those limits.
- The overlay prefetches any hash named in an SCP envelope before Core
checks the signature, so a peer can buy one frame over the rate per
envelope; a reconnect resets a peer's allowance; many colluding peers
can even out the shared byte budget; the 100-entry cache itself is first
in, first out over all sets.
- Hardening of the overlay's XDR parsing limits is not addressed here;
it is the same on the base for every message type.

## Testing (all on `8f7768be7`)

- New, C++ (`herder/test/TxSetPushTests.cpp`, 4 network simulations with
real overlay processes): the leader's push is used (no follower waits
for the retry); consensus through the request path when no push is sent;
a network where half the leaders push; every leader pushes one set and
proposes another (the pushed set is never used, every ledger
externalizes the proposed set). One IPC section in
`overlay/test/IPCTests.cpp`.
- New, Rust: 16 unit tests for the push bookkeeping
(`flood/txset_push.rs`), 7 two- and three-node tests
(`libp2p_overlay/txset_push_tests.rs`: push reaches every peer once, a
crossing request is skipped once, a late request is answered, a peer
that missed the push is served, the record is cleared on a closed
connection, the rate limit, a malformed frame), 7 App-level tests
(`main.rs`: a pushed set reaches Core only on request, a set with
another hash does not answer a request, the limits), 4 end-to-end tests
with two overlay binaries (`tests/e2e_binary.rs`: push before the
request, no push, pushed set with another hash, a dropped push recovered
by a request).
- Full suite: PASS. 602 test cases / 2,383,553 assertions in 121
batches, libsodium 77, xdrpp 7, check-rust and the soroban host tests
(103 `test result: ok`, 0 failed).
- Full suite `--all-versions`: PASS. 602 test cases / 8,155,400
assertions.
- Tag run
`[herder],[scp],[txset],[overlay-ipc],[overlay],[ipc],[upgrades],[simulation]`
(150 tests, on the PR head `d0fe62652`): passes except three
`[acceptance]` tests that are not in the full suite and fail without
this change: "hierarchical topology scales 1 to 3" and "core nodes with
outer nodes" (as reported in #5495), and "SCP Driver / validateValue
txSet cached" (`HerderTests.cpp:2964`, `0 == 9`), which fails on #5495's
source too whenever the test runs with `--base-instance` 50 or 400, i.e.
in partition 2 or 9 of a partitioned run, and passes with 0 or 450.
- `clang-format` 20.1.8 on the changed C++ files and `cargo fmt --check`
on the overlay crate: clean.
- **Not passing:** a `--disable-tests` build stops in
`herder/HerderImpl.cpp` on the same test headers as on #5495. The three
other non-test objects this change touches compile without
`BUILD_TESTS`.
- **Not run:** sanitizers; the tx-meta and LedgerCloseMeta determinism
checks (the change does not touch apply).
- **Test gaps:** no test with a peer that really stops reading its
tx-set stream; none that drives a failed push write end to end; the C++
simulations run with empty mempools, so the prepared-set push and
realistic set sizes are only exercised on perfnet; with three or more
nodes, the path "request skipped, then Core's retry to another peer" is
not tested; the network tests use fixed ports and 1 s windows.
- Three independent review passes of the diff; the last one found no
correctness, determinism or safety problem. Its minor findings are under
"Open review findings".

## Perfnet

30 validators in 4 regions, 10 tier-1 orgs, c6id.2xlarge, SAC payments,
150 ms simulated apply per ledger (50 us per op at this setting), 900 s
measured, all SCP timeouts 2 s. Target 1,000 ms, 3,000 TPS. Base:
#5495's commit `daa965f4c` (before its rebases, which changed
formatting, the README and a CI file), build #3674, the same fixture and
instance type on the same day; its load was stopped after the first
window, as block_time_check does on a miss. This PR: build #3675 of
`8f7768be7`. **One run each.**

| | #5495 `daa965f4c` | This PR `8f7768be7` |
|---|---:|---:|
| Windows | 1 of 1 missed (all 30 nodes) | **4 of 4 on target (all 30
nodes)** |
| avg p75 per window | 1,386 ms | 1,091 / 1,096 / 1,109 / 1,115 ms |
| avg p99 per window | 1,599 ms | 1,227 / 1,243 / 1,253 / 1,290 ms |
| Ledgers per 5 min | 225-226 (1.33 s each) | 285-286 / 283-284 /
281-282 (1.05-1.07 s) |
| Txs per ledger | 3,750 (the cap) | 3,155 (below the cap) |
| Tx-set fetch `scp.fetch.txset`: mean (p75 / p99) | 0.217 s (0.33 /
n/r) | **0.016 s (0.017 / 0.024)** |
| Sets a follower had to await from the network | every set (0.198 s
mean) | none in window 1 |
| Trigger to first ballot: mean (p75 / p99) | 0.70 s | 0.42 s (0.47 /
0.57) |
| Ballot to externalize: mean (p75 / p99) | 0.41 s (0.53 / 0.76) | 0.33
s (0.34 / 0.50) |
| Leader build / follower validation / ledger close (mean) | 0.25 / 0.12
/ 0.20 s | 0.15 / 0.11 / 0.16 s |
| **Trigger to close (sum)** | **1.30 s** | **0.91 s** |
| Leader timeouts, prepare timeouts, rounds above 1, empty tx sets | 0,
0, none, 0 | 0, 0, none, 0 |
| Error counters, in sync, no fork, still closing | passed | passed |
| Inclusion | skipped (early stop) | **failed: 2,880,060 on every node,
2,880,000 offered** |
| Verdict as judged | FAIL (block time) | FAIL (inclusion only, with the
exact check; passes the relaxed one) |

Stage values are 5-minute means at the end of window 1, median over the
30 nodes.

- Push counters over the whole load (network totals): 28,507 pushes
sent, 28,456 received, 28,181 already in place when Core asked, 64
requests that crossed a push (not answered twice), 0 dropped, 0 evicted,
0 retries, 0 write errors.
- From live logs of one node per region: a follower's Core has the set a
median 126 ms after the leader logs "Built TX set" (76-169 ms by region
pair). On #5495 that took about 480 ms on a 147 ms link. A follower
externalizes 0.45-0.55 s after the leader's set is built (0.97 s on
#5495 at the 2 s setting).
- The two runs are in different regimes. #5495 could not close a ledger
in under 1.33 s, so every set was at the cap and the mempool backed up.
Here the cycle fits into about 0.91 s and ledgers hold what was offered.
That also makes stages this change does not touch cheaper (the leader
pulls 3,200 candidates instead of 7,700; sets are smaller).
- Block time is about 1.05 s although the stages sum to 0.91 s; not
looked into.
- A slow drift over the 16 minutes: avg p75 1,091 to 1,115 ms, trigger
to first ballot 0.42 to 0.45 s. Cause not identified.
- **Not separable in one run:** the push and the 8 MiB window. The
window also applies to SCP and transaction flooding.

### The inclusion check: 60 transactions too many

block_time_check compares the transactions in every node's ledgers with
the number offered. All 30 nodes hold exactly 60 more (same value
everywhere, no fork). The generators submitted exactly 2,880,000 (96,000
each, none rejected) and every node pulled every foreign transaction
exactly once, so about 60 transactions (0.002%) were included in two
ledgers. #5495's completed run at 2 s had the exact count; its 1 s run
stopped before this check.

How that can happen here (the code path is the stack's documented
behaviour; that it explains these 60 is inferred from the counters, the
logs carry no transaction hashes):

1. The fixture runs in overlay-only mode. Apply is skipped, sequence
numbers do not advance, and the one-transaction-per-source-account rule
is off, so nothing in Core rejects a transaction that an earlier ledger
already held. With real apply such a copy fails validation when a leader
builds its set.
2. The overlay's mempool removes a ledger's transactions at
externalization and deliberately keeps no memory of them: the test
`externalization_removes_queued_admission_without_remembering_hashes`
says "Truly later admissions are deliberately allowed; no finalized hash
history remains. Ledger-state validity is still checked by Core." A copy
that reaches a node for the first time after the removal is admitted as
new and included when that node next leads.
3. A leader that is not backed up takes everything in its mempool,
including transactions that are still flooding to the far nodes (an
inventory batch plus one and a half round trips: up to about 0.5 s).
4. With the push, a follower externalizes 0.45-0.55 s after the leader's
set is built, which is now as fast as the slowest flooding. On #5495 it
was about a second, so the two never met.

So the design relies on Core's ledger-state validation, which this
fixture switches off, and the push's speed-up is what makes the two
meet. This PR does not change that, and the overlay keeps its behaviour
(remembering finalized hashes there would change production behaviour
for a test-mode artifact). The duplicates are accepted for this fixture:
block_time_check's inclusion check is relaxed to allow up to 1 in 10,000
transactions more than offered, with the caveat written next to it
(stellar-experimental/performance-testing#4). Fewer than offered, more
than that, and nodes that disagree still fail. No run has been made with
the relaxed check; applied to this run's recorded counts it passes with
60 over.

## Open review findings (not fixed in `8f7768be7`)

- The in-flight count of a push is only released when the write task
ends. A panic in that task, or a stream open that never completes while
the peer is disconnected, would leave the peer marked busy (it would
then always request).
- A push that is still being written when a connection to that peer
closes is no longer recorded: a crossing request gets a second copy, and
a request skipped before the close is not answered if the write then
fails (Core's retry covers it).
- Per-peer unclaimed entries can add up to 20 while the cache holds 100
sets of any kind.
- A set is accounted to the peer that delivered it first, so a peer that
delivers the leader's set first and then pushes four more evicts it.
- The token bucket's parameters assume at most two sets per led slot at
500 ms or more per slot.

## Next steps

Not started: a run of the base with only the congestion window, and of
the push on quinn's default window, to separate the two effects; repeats
(n = 1 here); a longer run to see whether the drift continues; then 800
ms.
SirTyson added a commit that referenced this pull request Nov 11, 2026
…peculatively, skip an invalid set at apply (#5498)

**Prototype, not for merge.** Stacked on #5497 (push the leader's tx
set). This PR adds two commits to #5497's four: `a33da62c4`, the change,
and `0c9a02e56`, a fix for a bug in the Core-overlay IPC reader that the
perfnet run below hit (it is in the stack below this PR).

From protocol 30, a validator votes for the round leader's value without
validating its transaction set. The validity check moves to ledger
apply, where an invalid set closes an empty ledger, and the ledger is
executed speculatively while consensus is still in flight. The
transaction set is made durable before the vote by the overlay, in its
compressed form. There is no compatibility work beyond the protocol
gate; the PR exists to measure what validation, persistence and apply
cost on the leader-schedule stack.

## Result in one paragraph

At the new target fixture (16 tier-1 orgs = 48 validators in 4 regions,
c6id.2xlarge, 1 s target, 5,000 SAC TPS, one run per build) #5497 alone
misses: 1.42 s per ledger, avg p75 1,461 ms, every tx set at the cap.
With this PR on top, all four windows are on target on all 48 nodes: avg
p75 1,094 to 1,137 ms, 1.04 to 1.07 s per ledger, and the time from
trigger to ledger close falls from 1.37 s to about 0.8 s. Every ledger
was executed speculatively and committed as executed (hit rate 100%, no
execution dropped, no set skipped). The run's verdict is still **FAIL**:
one node dropped out of consensus 15 s before the load ended, because
Core's IPC reader thread took an interrupted `recv` for a closed
connection. That bug is in `src/overlay/IPC.cpp`, below this PR; the
second commit fixes it, and a rerun with the fix is in progress. The
comparison is between two regimes (over capacity against keeping up),
and the speculation gain on this fixture is an upper bound (see
Perfnet).

## What changes (all gated on protocol 30; protocol 29 and older are
unchanged)

- **Vote without validating.**
`HerderSCPDriver::validateValueAgainstLocalState` returns
`kFullyValidatedValue` for a value whose tx set the node holds, without
running the set's validity check. Still checked before the vote: the
StellarValue itself (signature, close time, upgrades), that the node
holds the set, and one O(1) property of the set, that it was built on
the LCL.
- **Validate at apply; an invalid set applies as an empty ledger.**
`LedgerManagerImpl::validateTxSetForApply` runs `prepareForApply` and
`checkValid` against the previous ledger with the value's close time. If
either fails, nothing of the set is applied: no fee, no sequence bump,
no result.
- The header keeps the externalized value (the invalid set's hash);
`txSetResultHash` is the hash of an empty result set.
  - Upgrades in the value still apply.
- History keeps the set although the ledger has no results, and
LedgerCloseMeta carries it with no transaction meta. A node replaying
the ledger needs the set to match the header, and reaches the same
verdict through the same code.
- This is not the existing empty-tx-set value: consensus runs on, and
externalizes, the invalid value.
- **Speculative execution.** `applyLedger` is split into `executeLedger`
(validation, fees, transaction execution; everything it does lives in an
object that rolls back when dropped) and `commitLedger` (history,
upgrades, seal into the BucketList and database, meta, LCL). When a
validator emits a PREPARE or CONFIRM for a value for LCL+1, the ledger
apply thread executes that ledger. When a value externalizes, the
executed ledger is committed if it is for the same ledger, value and
set; otherwise it is dropped and the ledger is applied from scratch.
`EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY` (default true) turns it off.
- **Persist the compressed set, in the overlay.** The Rust overlay
writes the zstd frame of every tx set Core holds to
`<BUCKET_DIR_PATH>/txsets` (temp file, `fdatasync`, rename) before it
delivers a set to Core, and for a set Core built before it handles
Core's next IPC message (the push and the proposal). Core no longer
serializes, base64-encodes and commits those sets to SQLite in front of
its first vote; after a restart it requests the sets its restored SCP
state names, and the overlay reads them back from disk. Only sets Core
holds are stored, and they are deleted by slot age, so peers filling the
overlay's cache cannot evict them. If a set Core built cannot be stored,
the overlay exits instead of sending anything that refers to it.
- The validity check no longer reads the LCL through the application: it
takes a `TxSetValidationContext` (previous header, Soroban config, a
factory for read-only ledger views), so it can run on the apply thread
against apply's own snapshot.
- New metrics: `ledger.apply.{txset-skipped, execute, commit,
speculation-started, speculation-hit, speculation-miss,
speculation-idle}`, `herder.scp.persist`. `herder.txset.validate` now
times the apply-time check.

## Why it should be safe, and where it is not

- **Consensus safety is unchanged**: nodes vote for more values than
before, never for two.
- **The apply-time verdict is a function of the previous ledger, the set
and the value's close time**, read from an immutable snapshot: the same
inputs the pre-vote check used. Live apply and replay run the same
function.
- **Speculation is not observable.** The same two functions run in the
same order whether execution was early or not. A test compares ledger
hashes for no speculation, speculation on the externalized value and
speculation on another value.
- **What gets worse, by design of the experiment:**
- Any nondeterminism in `checkValid` used to cost liveness at most. It
now forks the ledger. Its node-local inputs are the binary's protocol
version, the network passphrase and, in test builds, loadgen flags
(`ARTIFICIALLY_GENERATE_LOAD_FOR_TESTING`, `LOADGEN_BYTE_COUNT_*`):
validators that differ in those would disagree on validity.
- A leader can get an invalid set externalized. The ledger is empty, and
the set's transactions are still removed from every mempool at
externalize, so a faulty leader can drop pending transactions at no cost
whenever it leads.
- Downstream consumers see new shapes: a LedgerCloseMeta with a
non-empty tx set and no transaction meta, and a history `transactions`
entry without a `results` entry.

## Testing

On `a33da62c4` (the change), devbox, 32 partitions:
- **Full suite: PASS.** 614 test cases, 2,384,306 assertions, plus
libsodium, xdrpp, check-rust and the Soroban host tests. 12 of them are
this PR's new cases and 4 are #5497's.
- **Overlay Rust tests: PASS.** 199 library, 44 binary, 7 end-to-end.
- **Full suite `--all-versions`: PASS.** 614 test cases, 8,126,148
assertions.

On the same change before the rebase (`25f48abb6`, tree of `3b8a8356c`):
- **tx-meta baseline** (`[tx]`, all versions, seed 12345): unchanged,
144 test cases.
- **LedgerCloseMeta golden data** (`--capture-lcm`): no file changed.

New tests:
- `herder/test/ApplyTimeTxSetValidationTests.cpp` (6 cases): an invalid
or malformed set closes an empty ledger and the next ledger applies
normally; upgrades of a skipped value apply; the same set applies or is
skipped depending on the value's close time; protocol 29 applies the set
as before; `validateValue` levels on both sides of the gate; a 4-node
network externalizes a leader's invalid sets and every node closes them
empty with the same hash; a node restarts and gets its tx set back from
the overlay's store, with no tx-set row in SQLite.
- `ledger/test/SpeculativeApplyTests.cpp` (5 cases): an executed ledger
is committed when its value externalizes; it is dropped for another set,
another close time, a newer speculation, an explicit abort, and at
shutdown; an invalid set under speculation; identical ledger hashes with
and without speculation, including upgrades and skipped sets; a 4-node
network where every node commits speculatively executed ledgers.
- `history/test/HistoryTests.cpp` (1 case): a node replays skipped
ledgers from an archive and closes the same hashes. It fails if history
does not keep the skipped set.
- Rust: `flood::txset_store` (4) and App tests (5): on disk before Core
gets the set; an unstorable set is not delivered, and stops the overlay
if Core built it; served to Core after a restart; deleted by slot age;
peers cannot evict a stored set.

Existing tests changed: four "SCP Driver" assertions and the
signature-prewarm test that encoded validation before voting; "network
does not externalize non-canonical empty tx set" is now "network
externalizes non-canonical empty tx set" (it externalizes and closes an
empty ledger).

On `0c9a02e56` (the IPC fix on top, this PR's head): **full suite
PASS**, 615 test cases, 2,385,530 assertions. The new test "IPC read
survives a signal" fails without the fix. `--all-versions` on this head
is running.

Not passing or not run:
- Tests outside the regular suite that fail the same way without this
change: "Catchup recent" and "resilience tests" (checked on
`overlay-v2-shared` `87201aa66`), "hierarchical topology scales 1 to 3"
(as recorded in #5495).
- One "SCP Driver" assertion (protocol 29, validity-cache hits) failed
once in a loaded partitioned run of the pre-rebase change and passed in
three runs alone. Not explained.
- A `--disable-tests` build was not tried; #5495 records that the base
does not build that way.
- No sanitizer run.

## Perfnet

16 orgs (48 validators), 4 regions, c6id.2xlarge, 1,000 ms target, 5,000
SAC TPS, 150 ms simulated apply per ledger (30 us per op), tx-set cap
6,250, 900 s measured, SCP timeouts 2 s. Same day, back to back, **n = 1
per build**. This fixture has no earlier comparison point.

| | #5497 alone (build #3675, `8f7768be7`) | This PR without the IPC fix
(build #3677, `a33da62c4`) |
|---|---:|---:|
| Verdict | **FAIL**: window 1 missed on 48/48 nodes; load stopped after
it | **FAIL**: node-45 out of sync and its load generator failed; all
four windows on target on 48/48 nodes |
| avg p75 per window | 1,461 ms | 1,094 / 1,116 / 1,130 / 1,137 ms |
| avg p99 per window | 1,638 ms | 1,313 / 1,453 / 1,408 / 1,435 ms |
| Ledgers per 5 min (per node) | 211-212 (1.42 s each) | 288-290, then
281-283 (1.04-1.07 s each) |
| Txs per ledger; sets at the 6,250 cap | 6,250; every set |
5,190-5,340; 5% of ledgers |
| Leader build (mean) | 0.411 s | 0.232-0.249 s |
| Trigger to first ballot (mean) | 0.801 s | 0.421-0.468 s |
| Ballot to externalize (mean) | 0.367 s | 0.296-0.298 s |
| Externalize to ledger close complete (core logs, mean) | 0.208 s |
0.071 s |
| **Trigger to close** | **1.37 s** | **0.79-0.84 s** |
| `herder.txset.validate` (mean) | 0.195 s, before the vote, main thread
| 0.178-0.190 s, at apply, apply thread, during the ballot |
| `ledger.apply.execute` / `ledger.apply.commit` (mean) | no such timers
| 0.33-0.35 s / 0.037-0.041 s |
| Speculation: started / hit / miss / sets skipped (network, whole load)
| n/a | 46,847 / 46,853 / 0 / 0 (hit rate 1.000 on every node) |
| Finished execution waiting for externalize (mean, median node) | n/a |
8-10 ms |
| `herder.scp.persist` (mean; p99), in front of every SCP broadcast | no
timer in this build | 2.2-2.6 ms; 5-6 ms |
| Leader and prepare timeouts, rounds above 1, empty tx sets | 0, 0,
none, 0 | 0, 0, none, 0 |
| Error counters / in sync / no fork / still closing | passed / passed /
passed / passed | passed / **node-45 not in sync** / passed / passed |
| Inclusion | check skipped (early stop); 6.8% short, over capacity |
check skipped (a generator failed); 4,798,690 of 4,800,000 on the 47
healthy nodes |

Reading it:
- **Two regimes.** #5497 alone is over capacity at this setting: 5,000
TPS need 7,075 transactions per 1.42 s ledger against a cap of 6,250, so
every set is full, the mempool backs up and every trigger is already
due. With this PR the network keeps up, sets are not full and the
leader's build is shorter for that reason alone (0.41 s to 0.24 s). The
stage differences are not all this PR's code.
- **What this PR's code accounts for:** followers no longer validate
(0.195 s) or write the set to SQLite before their first vote, and apply
after externalize shrinks from 0.21 s to the commit (0.04 s) plus
whatever of the execution is still running (0.07 s in total from the
logs).
- **The speculation gain is an upper bound.** The simulated apply is a
sleep, so running it during the ballot costs no CPU. Real execution
would compete with SCP processing and the signature prewarm.
- **The sleep is not placed the same way in the two builds.** In the
base it covers everything from the start of apply (commit included).
Here it sits in the execute phase, and validation and commit come on
top.
- **Executions finish with little to spare.** The validation plus the
simulated execution take 0.33-0.35 s against about 0.3 s of ballot; a
finished execution waited 8-10 ms on average for its value to
externalize.
- **The push from #5497 loses its race here.** In the base run 13,914 of
14,352 pushed sets were in place when Core asked. In this run Core's
request reached the leader before the push had arrived in 38,668 cases
against 7,032, because the proposal now leaves earlier relative to the
push (no SQLite write in front of it). It cost about 0.02 s per ledger
(`scp.fetch.txset` 0.052 s against 0.033 s); no retry was needed.

**The failure.** node-45 stopped following consensus at ledger 1024, 15
s before the generators finished; node-23 and node-32 did the same two
minutes after the load. On each, Core logged `Overlay IPC connection
closed` a few milliseconds into a checkpoint publish while both
processes stayed up and the socket stayed open, with the overlay's send
queue full. `readExact` in `src/overlay/IPC.cpp` returns "closed" for
any `recv` result <= 0, `EINTR` included, and stellar-core's signal
handlers do not restart system calls; the inference (no errno was
logged) is a SIGCHLD from the publish's child processes delivered to the
reader thread. The base run had no such event in 30 minutes, this run
three in 40: one run each cannot say whether this PR makes it more
likely. None of this PR's own error paths fired on any node. Commit
`0c9a02e56` retries on `EINTR` and adds a test that interrupts a blocked
reader.

A rerun of this PR's head with that fix (build #3678) is in progress;
the table gets a third column from it.

## Review findings not fixed

An independent review of the pre-rebase change found no case where the
speculative path commits a different ledger than the regular one, or
where live apply and replay decide differently. Not fixed:
- The two "gets worse" items above (node-local test flags in validity;
mempool purge of a skipped set).
- A `CacheTxSet` that the overlay drops for a hash mismatch or an encode
error (both Core bugs) leaves Core believing the set is stored.
- A value is not speculated again after its execution was dropped;
`ledger.ledger.close` leaves out the time a finished execution waits and
executions that were dropped; in test builds
`getLastClosedLedgerTxMeta()` can show a ledger that is still
speculative.
- Test gaps: the hash-equality test covers classic payments and upgrades
only (no Soroban parallel apply, no DEX offers, no meta comparison);
speculation tests always let execution finish before externalizing or
aborting; no kill-restart test of the store.
- Open question: a parked execution keeps a SQL transaction open for the
ballot window. Not confirmed that no other writer on the main database
can make its commit fail.
- The merge with #5497 (a pushed set is stored before Core gets it, like
any other) was not reviewed separately.

## Next steps

Not started. The commit and the next leader's build still wait for
externalize. Letting the next leader build on the speculatively executed
state is the next 0.2 s or so, and needs either every node pre-building
or a schedule seed that is known one ledger earlier.
SirTyson added a commit to SirTyson/stellar-core that referenced this pull request Nov 11, 2026
… test helper

It fails for one combination of the day's test seed and a partition's
instance number, also on the binary of stellar#5495 alone.
SirTyson added a commit that referenced this pull request Nov 11, 2026
… validation results (#5499)

README only, no code change.

The benchmark for this branch is now **48 validators (16 organizations,
4 regions), 5,000 SAC TPS, 1,000 ms target**. The README reported the
30-validator / 3,000 TPS runs of #5495 and #5497; this updates it to the
runs at the new fixture and adds #5498.

## What the README now says

| | Tx-set push alone (#5497, `8f7768be7`) | With apply-time validation
(#5498, `0c9a02e56`) |
|---|---|---|
| Block time windows | missed the first window on all 48 nodes | on
target in all four windows, on all 48 nodes |
| p75, average over 48 nodes | 1,461 ms (limit 1,200) | 1,081 / 1,132 /
1,118 / 1,142 ms |
| Time per ledger | 1.42 s | 1.03-1.08 s |
| Trigger to ledger closed | 1.37 s | 0.76-0.84 s |
| Run verdict | FAIL (block time) | FAIL (inclusion check only) |

- **The run with the change fails the inclusion check, and only that
check.** Every node holds 4,800,060 transactions against 4,800,000
offered. The check is an exact comparison (performance-testing#4, which
would allow this, is not merged).
- **The 60 extra transactions are not explained.** The 30-validator run
of #5497 had exactly 60 extra too. The explanation given then (late
copies included twice in overlay-only mode) was an inference, and it
does not predict the same number at a different rate and network size.
The README now says so.
- **The IPC bug.** The change was first run without its last commit
(`a33da62c4`): same block times, but one node stopped reading its
overlay IPC socket 15 s before the end of the load. The fix
(`0c9a02e56`) is in the reported run, where fourteen sweeps of all 48
nodes (44 checkpoint publishes per node) found no affected node. One
clean run does not prove the fix; the "before" run had no such event
either.
- One run per build. The gain from speculative execution is an upper
bound, because apply is simulated by a sleep.

## Changes

- Title and intro: three experiments; the benchmark fixture.
- "What this branch is": adds experiment 3 (vote without validating,
validate at apply, empty ledger for an invalid set, speculative
execution, tx set stored by the overlay; protocol 30 gate).
- "Performance results": the 48-validator runs, windows, stage table,
speculation and tx-set store numbers, the inclusion check, the IPC bug,
where a ledger's time goes now.
- The 30-validator / 3,000 TPS results are condensed into one section
that links to the README as of #5497 for the details.
- "How it was measured": 16 organizations, 12 nodes per region, full
mesh of 47 peers, tx-set cap 6,250, 30 us simulated apply per operation,
the command with `--org-count 16`.
- "Known limitations": adds the open items of #5498.
- Test status of the branch head, including one `--all-versions` failure
that comes from a test helper of #5495 (it fails for one combination of
the day's test seed and a partition's instance number, also on the
binary of #5495 alone).

## Corrections to the previous README

- It said the perfnet inclusion check "has since been relaxed". That
change is not merged.
- It presented the cause of the 60 duplicate inclusions as understood.
It is not.

## Not verified

- The quorum threshold (11 of 16 organizations) is computed from
stellar-core's rule for generated quorum sets, not read back from a
node.
- Round-trip times between regions are the ones measured on the
30-validator fixture.
SirTyson added a commit that referenced this pull request Nov 12, 2026
… an admission check (#5501)

**Prototype, not for merge. Stacked on #5502** (leader terms,
pre-validation during the previous ballot, one build before the
trigger): this PR's own changes are the last two commits, `6ab837c27`
(the code) and two README commits (the results of the whole stack); the
commits before them are #5502's (head `8aa7f6bf0`) and leave this diff
when #5502 merges.

A node that accepts a transaction from a client checks it against its
last closed ledger, keeps it, and has its overlay send the body straight
to the leaders of the current and the next round of the next ledger,
instead of advertising it to every peer (INV/GETDATA). Since protocol 30
a follower votes without validating, so it does not need a transaction
before the leader's set arrives.

## Results in short

One run per configuration everywhere; perfnet `block_time_check`, SAC
payments, 150 ms simulated apply, overlay-only mode.

**On top of #5502, 16 tier-1 orgs (48 validators in 4 regions), 800 ms
target, 8,000 TPS, c6id.8xlarge (16 cores):**

- **Block time is the same as #5502 alone, on target in all four
windows:** avg p75 804 to 805 ms against 805 to 807 ms; trigger to close
0.43 to 0.44 s against 0.44 to 0.45 s. Both runs sit at the fixture's
floor, so this run shows neither a gain nor a loss in block time.
- **Overlay traffic per node falls from 21.2 to 6.2 MB/s each way
(-71%).** No adverts or demands are left; 3.3 transaction bodies are
sent per offered transaction.
- The leader's side is unchanged: one build per ledger before the
trigger, the pre-validation covers 66.7% of the candidates (69% on #5502
alone), leader terms of exactly 5 ledgers, no timeout.
- Followers verify signatures inline: the apply-time check takes 0.099 s
against 0.081 s, still inside the ballot on 16 cores.
- The verdict is FAIL on the inclusion count only, as for #5502 alone
(the check is exact; see "Inclusion" under Perfnet).

**On top of #5502, the goal fixture: 30 orgs (90 validators), 800 ms,
10,000 TPS, c6id.8xlarge (this PR's head):**

- **FAIL: window 1 at avg p75 980 ms (limit 960), p99 1,441 ms (inside
its 1,600 limit); the load was stopped after it.** #5502 alone at this
fixture: 1,032 ms and p99 1,887 ms.
- A ledger every 0.90 s against 1.02 s; 39% of the sets at the 10,000
cap against 93%; three of four leader builds before the trigger against
none; trigger to close 0.77 s against 0.97 s. The gain is in trigger to
first ballot (0.39 s against 0.58 s): the leader's mempool holds about
one ledger of candidates instead of a 20,200 backlog.
- **Overlay traffic per node: 15.7 MB/s against 43.** At 90 validators
most of what is left is SCP (35,700 messages per second per node).
- What still does not fit into 800 ms is the ballot (0.32 s), the set
reaching followers 0.17 s after the proposal, and the builds that still
run at the trigger. None of these is transaction dissemination.
- **Where the remaining time goes** (twelve nodes' logs): the leader's
build starts when the previous ledger closes and takes 0.29 s to the
proposal; the set then needs 0.19 s to reach a follower (the leader
sends 5.7 MB of XDR, compressed, to each of 89 peers); the ballot takes
0.34 to 0.36 s; the commit 0.07 s. Sum 0.89 s on the leader. The README
has the table.

**Before the rebase (on `overlay-v2-shared` at `26bea1150`, without
#5502), 16 orgs, 1 s target, 5,000 TPS:**

- Overlay traffic per node fell from 14.0 to 4.7 MB/s each way (-66%) in
every run.
- Whether the block time held depended on validation CPU: FAIL with the
default 3 validation tasks on 4-core c6id.2xlarge (avg p75 1,367 ms;
base 1,081 ms), PASS with 7 tasks on the same instances (1,030 ms, every
check including an exact inclusion count), all windows on target on
16-core instances (1,019 to 1,021 ms; inclusion +250).
- At 30 orgs the base failed 1 s / 5,000 TPS (1,298 ms); this change was
not run there.

## What changes

- **One function decides who gets transactions**:
`HerderImpl::computeTxFloodTargets()`. With the leader schedule: the
leaders of the node's current round and of the next one for the slot
after the LCL (two nodes). With #5502's leader terms the node that
closes a ledger leads the next one too; one slot before a new term
starts (every fifth ledger), that term's expected first leader is a
third target, so that it holds the transactions when it pre-validates
and builds its first set. It is the first node of the order seeded with
the LCL's hash, so it is known a full ledger ahead. "Everyone" (the old
flood) when the node is not tracking or more than one ledger behind,
when there is no schedule, and for a pre-protocol-20 LCL. It is
re-evaluated at a ledger close, at a round change, at an externalize and
at loss of sync, and sent to the overlay when it changes (IPC
`SET_TX_TARGETS`). A changed leader schedule only has to change this
function.
- **The accepting node keeps the transaction** (overlay
`flood/tx_router.rs`, at most 100,000 transactions and 300 s) until it
is externalized, removed by Core or expires. When the targets change, it
goes to a newly added target only if it may still be wanted there: it is
younger than 200 ms, or three ledgers externalized on this node without
it since it was accepted, or it is older than 3 s (no ledger closes).
Anything else is most likely in the set in consensus. Nothing is sent
again to a peer that got it less than 60 s ago on the same connection; a
failed send or a reconnected peer is served regardless. A node that
received a transaction as a target keeps it in its mempool and does not
relay it.
- **Fallback to the flood.** If a target is not a directly connected,
identified peer, the node floods its transactions with adverts as
before, so they reach every mempool whatever the topology. A node that
got a transaction directly joins a later flood of it and can serve the
body.
- **Admission check** (`HerderImpl::checkTxAdmission`, in
`recvTransaction`): `checkValid` against the LCL, the check the leader's
build runs on every candidate, and one pending transaction per source
account per accepting node (`TRY_AGAIN_LATER`; `DUPLICATE` for the same
hash). On the base `recvTransaction` checked nothing. The per-account
record goes when a set that uses up the sequence number closes, when the
LCL shows the sequence number used some other way, or after 300 s.
- **Overlays prove which validator they belong to.** The overlay's
libp2p identity is a random key. At startup it sends its peer id to
Core; Core signs `SHA256(networkID || "stellar-overlay-peer-id" ||
peerId)` with the node key; the overlay shows `[nodeID][signature]` to
each peer on a new stream (`/stellar/node-id/1.0.0`, acknowledged,
retried), and the receiver checks it against the peer id QUIC
authenticated. The node key stays in Core.
- **No leader prewarm of its own any more.** The version on the old base
verified arriving signatures between the close and the trigger. #5502's
leader builds once before the trigger and pre-validates during the
previous ballot; bodies that arrive after those pulls are verified in
the build, as on #5502 alone.
- **Options, for measurements:** `TX_FLOOD_TARGET_LEADERS` (default 2; 0
= every validator of the schedule, still direct and without adverts),
`TX_FLOOD_LEGACY_ADVERTS` (the base's flood from the same binary),
`BATCH_EXECUTOR_TASKS` (number of validation tasks; above the physical
core count they run on hyper-thread siblings).
- New metrics: `overlay.flood.tx-direct-sent`, `.tx-direct-sent-bytes`,
`.tx-direct-recv`, `.tx-local-flooded`, `.tx-late-copy` (a transaction
admitted although one of the last few externalized ledgers already has
it), `overlay.node-id.identified`, `.rejected`,
`herder.tx-admission.check`, `.invalid`, `.account-busy`.

## Why ledger outputs cannot change, and what does change

- The change decides which transactions reach a leader's mempool, when,
and which submissions are refused. Building a set from a mempool,
validating it, voting and applying are untouched: a given tx set
produces the same ledgers and meta.
- The admission check reads the LCL on the main thread and writes
nothing.
- The identity binding is domain-separated from everything else the node
key signs.
- **What gets worse, by design:**
- Followers no longer hold transactions before the set arrives, so the
background signature verification finds nothing and the apply-time check
verifies signatures inline (see the results).
- A term's first leader is predicted one ledger ahead. If the ledger in
between costs that node its turn (#5502's forfeit), another node leads
and gets the transactions only at the close. #5502 also lets the second
node of the expected schedule pre-validate; it gets the transactions
when the ledger before closes, not earlier.
- A predicted target that is not a connected, identified peer makes the
node flood for the whole ledger before the term.
- A restarted node forgets the transactions it accepted and had not seen
included; on the base every node held them.
- A transaction submitted through a node that cannot reach both targets
directly costs what it costs today.
- **Left open:**
- A peer can still fill a target's mempool with invalid transactions
that outbid valid ones (the base's exposure, minus the relay). The
leader validates when it builds and drops them then.
- A refused submission returns `{"status":"ERROR"}` without the result
code.
- A pending transaction that dies for a reason other than its sequence
number (expired time bound) holds its account on that node for up to 300
s.
- Fee bumps are accepted and then dropped by the overlay, as on the
base.
- No protocol gating and no version negotiation: a node of the base
ignores the identity stream, and nodes with this change flood toward it.

## Perfnet

### On top of #5502: 16 orgs, 800 ms, 8,000 TPS, c6id.8xlarge

`block_time_check` (perfnet `btc2-slop`), `--org-count 16
--block-time-ms 800 --soroban-tps 8000 --soroban-type sac --classic-tps
0 --simulate-apply-budget-ms 150 --load-duration-s 900 --instance-type
c6id.8xlarge`, 48 validators in four regions, no log saving. The #5502
column is that PR's own run of the same day at the same fixture. The
stack column is build #3685 of `6b3c7ae0e`, which is this PR's head
**without** the changed re-send rule, without the late-copy meter and
without #5502's last commit.

| Windows 1 / 2 / 3 / 4 | #5502 alone (build #3684) | This PR on #5502
(build #3685) |
|---|---:|---:|
| Windows on target | 4 of 4 | **4 of 4** |
| Avg p75 block time, ms | 806.9 / 805.6 / 805.3 / 805.3 | **804.9 /
804.1 / 803.8 / 803.9** |
| Avg p99, ms (limit 1,600) | 1,153 / 1,188 / 1,100 / 1,249 | 1,170 /
1,209 / 1,039 / 1,172 |
| Txs per ledger; sets at the 8,000 cap | 6,450; 3.8% | 6,440; 3.4% |
| **`overlay.byte.read` per node** | **21.2 MB/s** | **6.2 MB/s (-71%)**
|
| `overlay.message.read` per node | 20,500 /s | 12,550 /s |
| Adverts per node | 7,848 /s | 0 |
| Direct sends per offered transaction | none | 3.3 |
| Leader build `herder.txset.build`; builds per ledger | 0.16 s; 1 |
0.15 s; 1 |
| Trigger to first ballot | 0.108 to 0.115 s | 0.100 to 0.118 s |
| Ballot to externalize | 0.29 s | 0.29 s |
| Externalize to close (core logs, mean) | 0.045 to 0.048 s | 0.041 s |
| **Trigger to close** | 0.44 to 0.45 s | **0.43 to 0.44 s** |
| Apply-time check `herder.txset.validate` | 0.080 s | 0.099 s |
| Signature-cache misses in that check, per node | (followers hold the
transactions) | 7,490 /s (44% of the lookups) |
| Validation-cache hit share in the leader's build | 69% | 66.7% |
| Leader terms | exactly 5 ledgers | exactly 5 ledgers, 225 terms, 48
leaders |
| Leader and prepare timeouts, empty sets | 0 | 0 |
| Inclusion, over the 7,680,000 offered | +80 | **+6,610** |
| Error counters, in sync, no fork, still closing | passed | passed |
| Verdict | FAIL on inclusion only | FAIL on inclusion only |

- The base `overlay-v2-shared` at this fixture on the same instances
(reported in #5502): window 1 on target at 846 ms, window 2 missed,
trigger to close 0.67 to 0.69 s.
- **Inclusion.** The check requires exactly the offered count. #5502
alone held 80 more, this run 6,610 more (0.086%): copies of transactions
that reach a node after their ledger closed and are included again,
which only overlay-only mode permits. This run's build re-sent every
pending transaction to newly added targets; the head sends only fresh
ones or ones that missed three ledgers, and the goal-fixture run with it
had 81 on 3,870,249 (0.002%) and counted 688 late copies
(`overlay.flood.tx-late-copy`). The 16-org fixture was not rerun with
the head.
- Every node identified all 47 peers, nothing was flooded, no send
failed, no submission was refused; the admission check takes 0.094 ms
per transaction on the main thread.

### On top of #5502: the goal fixture, 30 orgs, 800 ms, 10,000 TPS,
c6id.8xlarge

The same command with `--org-count 30 --soroban-tps 10000`; 90
validators, 10,080 TPS offered. The stack column is build #3688 of this
PR's head `6ab837c27`. The #5502 column is that PR's own run of its head
`8aa7f6bf0` two hours earlier (some rows are from its first run at this
fixture, marked G).

| Window 1 (the load was stopped after it in both runs) | #5502 alone
(build #3686) | This PR on #5502 (build #3688) |
|---|---:|---:|
| Avg p75 block time, ms (limit 960) | 1,031.6 | **979.6** |
| Avg p99, ms (limit 1,600) | 1,887 | **1,441** |
| Time between ledgers | 1.02 s | 0.90 s (median 0.88 s) |
| The first minute of the load | 802 ms for 30 s, backlog from 47 s |
805 ms over 60 s |
| Sets at the 10,000 cap | 93% | 38.7% |
| Leader builds before the trigger | 0 of 40 | 40 of 55 |
| Leader build; candidates pulled | 0.33 s; 20,200 | 0.24 s; median
8,350 (max 20,200) |
| Trigger to first ballot | 0.58 s | **0.39 s** |
| Ballot to externalize | 0.32 s | 0.32 s |
| Externalize to close | 0.07 s | 0.065 s |
| **Trigger to close** | 0.97 s | **0.77 s** |
| Set at the followers after the proposal | +230 ms (G) | +174 ms |
| Apply-time check `herder.txset.validate` | (see #5502) | 0.144 s |
| Validation-cache hit share in the leader's build | 66.3% (G) | 66.7% |
| **`overlay.byte.read` per node** | **43 MB/s** (G) | **15.7 MB/s** |
| Direct sends per submitted transaction | none | 2.5 to 2.6 while
keeping up, 3.7 in the backlog |
| Leader terms; timeouts | 5 ledgers; none | 5 ledgers; none |
| Submitted / included at the end | 49,855 short | 81 over (backlog
drained) |
| `overlay.flood.tx-late-copy` | n/a | 688, on 14 of 90 nodes |
| Error counters, in sync, no fork, still closing | passed | passed |
| Verdict | FAIL, window 1 (p75 and p99) | FAIL, window 1 (p75 only) |

**Not measured:** the 16-org fixture again with the head's re-send rule;
the base `overlay-v2-shared` at the goal fixture; the 4-core instances
with #5502 underneath; `TX_FLOOD_TARGET_LEADERS=0`; real apply; more
than one run of anything.

### Before the rebase: 16 orgs, 1 s, 5,000 TPS (on `overlay-v2-shared`
at `26bea1150`, without #5502)

All runs: `block_time_check` (perfnet `btc2-slop @ 6dd2064`),
`--block-time-ms 1000 --soroban-tps 5000 --soroban-type sac
--classic-tps 0 --simulate-apply-budget-ms 150 --load-duration-s 900`,
validators in us-east-1, us-west-2, eu-central-1 and ap-southeast-1,
every validator generates load, overlay-only mode (Core does not apply;
execution is a sleep). 5-minute means at the end of window 1, median
node.

| | Base | This PR, 3 tasks | This PR, 7 tasks | This PR, 16 cores |
Base, 30 orgs |
|---|---:|---:|---:|---:|---:|
| Build (Jenkins; commit) | #3678; `0c9a02e56` (tree of `26bea1150`) |
#3680; `dc91c8964` | #3683; `664a35fdc` | #3680; `dc91c8964` | #3678 |
| Validators x instance type | 48 x c6id.2xlarge | 48 x c6id.2xlarge |
48 x c6id.2xlarge | 48 x c6id.8xlarge | 90 x c6id.2xlarge |
| Validation tasks | 3 | 3 | 7 (`BATCH_EXECUTOR_TASKS=7`) | 15 | 3 |
| Verdict | FAIL on inclusion only (+60) | **FAIL**, window 1 | **PASS**
| FAIL on inclusion only (+250) | **FAIL**, window 1 |
| Windows on target | 4 of 4 | 0 of 1 (load stopped) | 4 of 4 | 4 of 4 |
0 of 1 (load stopped) |
| Avg p75 per window, ms | 1,081 / 1,132 / 1,118 / 1,142 | 1,367 | 1,030
/ 1,030 / 1,031 / 1,031 | 1,019 / 1,021 / 1,020 / 1,021 | 1,298 |
| `overlay.byte.read` per node | 13.97 MB/s | 4.77 MB/s | 4.71 MB/s |
4.77 MB/s | 26.69 MB/s |
| `overlay.message.read` per node | 14,804 /s | 9,374 /s | 9,914 /s |
10,085 /s | 33,523 /s |
| Direct sends per offered transaction | none | 9.0 | 3.4 | 3.1 | none |
| Leader build `herder.txset.build` | 0.235 s | 0.374 s | 0.099 s |
0.063 s | 0.314 s |
| Trigger to first ballot | 0.413 s | 0.539 s | 0.300 s | 0.232 s |
0.613 s |
| Ballot to externalize | 0.296 s | 0.290 s | 0.293 s | 0.284 s | 0.387
s |
| Apply-time check `herder.txset.validate` | 0.179 s | 0.277 s | 0.168 s
| 0.078 s | 0.247 s |
| Signature-cache misses in that check | 2,089 /s | 4,550 /s | 4,654 /s
| 4,647 /s | 2,841 /s |
| Externalize to close (core logs, median) | 58 ms | 145 to 210 ms | 44
ms | 28 ms | 61 to 102 ms |
| Ledgers per 5 minutes; txs per ledger | 291; 5,152 | 266; 5,637 | 300;
5,001 | 300; 5,000 | 262; 5,742 |
| Error counters, in sync, no fork, still closing | passed | passed |
passed | passed | passed |

- The base column is run 3 of #5498's record (same fixture, same box,
the evening before); it was not rerun.
- **3 tasks:** 75% of the sets were at the 6,250 cap (1.27 s per ledger)
and 16% were starved (as few as 938 transactions): after a slow ledger
the next trigger is already due and the new leader builds before the
pending transactions reach it.
- **7 tasks and 16 cores:** no starved sets, no backlog; the leader
pulls about 5,000 candidates at its trigger.
- Every run of this PR: every node identified all 47 peers as validators
(0 rejected claims), nothing was flooded, no message was dropped, no
submission was refused; the admission check takes about 0.12 ms per
transaction on the main thread.
- **Inclusion.** The check requires every node to hold exactly the
offered transactions. Base: +60; 16-core run: +250; 7-task run: exact.
In overlay-only mode Core's sequence-number check is off, so a copy of a
finalized transaction that a node admits again is included when that
node next leads (performance-testing#4, not merged, would allow up to
480 here). Which transactions were duplicated in the 16-core run, and
why more than on the base, is **not established**.

**Not measured before the rebase:** this change at 30 orgs; the base
with 7 tasks; real apply (with real execution the check and the
execution compete for the same cores).

## Testing

On `6ab837c27398` (this PR's head, on #5502 at `8aa7f6bf0`), devbox, 32
partitions:
- **Full suite: PASS** (exit 0, 718 s), with libsodium, xdrpp and the
overlay's Rust tests (224 library, 47 binary, 7 end-to-end; 4 ignored).
- **Full suite `--all-versions`: PASS** (exit 0, 735 s).
- Tag run
(`[herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[simulation]~[.],[loadgen]~[.]`,
which includes #5502's tests): only the three failures recorded for the
base ("SCP Driver" validity-cache hits, "hierarchical topology scales 1
to 3", "core nodes with outer nodes").
- **One unexplained Rust failure:** on an earlier head (`6b3c7ae0e`) one
`make check-rust` run failed 1 of 219 library tests and the output was
not kept, so the test is not known. About twenty later runs of the same
suite passed (apart from failures of tests I was changing at the time).
#5502 does not touch `overlay/`.

New tests:
- `herder/test/TxFloodTargetsTests.cpp` (6 cases):
- In a 4-validator mesh every submitted transaction is applied on every
node with no advert and no demand, and every target list each node sent
to its overlay equals what the schedule functions give for the LCL and
round it was sent at: two round leaders, and a third node only in the
ledger before a new term. A list sent one ledger before a term start
names the node that then leads the term, unless the ledger in between
forfeited. **Mutation-checked** three ways: the router forced to flood
(96 adverts for 32 transactions), the term lookahead removed, and the
lookahead seeded with the wrong hash; each fails the test.
- With `TX_FLOOD_TARGET_LEADERS = 0` each transaction goes to the three
other nodes exactly once.
- With one of three validators down every transaction is still included
and every target list matches the schedule at the node's round,
including rounds after a timeout.
- In overlay-only mode with load from all four nodes, no transaction
hash appears in two ledgers (2,400 transactions; it does not reproduce
the perfnet duplicates, which need network latency).
- The admission check (accepted once per account, duplicate, bad
signature, used and future sequence number, low fee, missing account,
freed by the including ledger, by another transaction of the account, by
a sequence bump, by timeout; overlay-only mode; pregenerated load); the
experiment options.
- Rust `flood/tx_router.rs` (20 unit tests): targets only and following
the targets; a kept transaction follows a target change only while
fresh, after three ledgers without it, or after 3 s
(**mutation-checked**: with the rule disabled five tests fail); a
removal by Core is not a ledger; flood without reachable targets; a
directly received transaction joins a later flood and serves the body.
- Rust `libp2p_overlay/tx_direct_tests.rs` (4 network tests on loopback,
1 unit test): the same over real connections, including a late copy
being counted; forged, replayed and wrong-network identity claims are
rejected, and claims are bounded per connection. `main.rs` (3): the IPC
handlers.

Existing tests changed:
- `overlay/.../dispatch_tests.rs` (2): they fed a body without a demand
and expected a relay; they now register the demand, and one asserts that
an undemanded body is not relayed.
- `main/test/CommandHandlerTests.cpp` "transaction envelope bridge": V1
and fee-bump envelopes are refused again below protocol 13, and the
too-early / too-late cases are back.

Review: four passes of an independent review (two on the old base, one
of the rebase, one of the re-send rule); no consensus or determinism
finding in any. Open from the last two: the flush after an externalize
plans against the outgoing targets (one stray body for transactions that
waited three ledgers); a lost send re-delivers to every held-back
target, not only the lost peer; after a leader failure the round-2
leader lacks transactions older than 200 ms until the ledger closes; the
200 ms bound is a constant, not derived from the block time; no test
covers a watcher, a wrong term prediction, a lagging sender or the
`main.rs` wiring of the ledger count.

Not passing or not run:
- Tests outside the regular suite that fail the same way on the base:
"resilience tests", "cycle4 topology", "asymmetric topology report
cost", and the three above.
- A `--disable-tests` build was not tried; #5495 records that the base
does not build that way.
- No sanitizer run. No tx-meta baseline run (the diff does not touch
apply).

## Commits

Three commits on top of #5502: the code, and two for the README (which
now describes and reports the whole stack: #5502's leader terms and
pre-validation, and this PR). Builds that perfnet ran: #3688 of the head
`6ab837c27` (the goal-fixture run), #3685 of `6b3c7ae0e` (the 16-org
run; that commit is no longer on the branch) and, before the rebase,
#3680 of `dc91c8964` and #3683 of `664a35fdc`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant