Skip to content

Select loadgen source accounts in constant time - #5458

Merged
SirTyson merged 5 commits into
pr/merge-ms-close-time-basefrom
pr/loadgen-account-selection
Nov 10, 2026
Merged

SirTyson merged 5 commits into
pr/merge-ms-close-time-basefrom
pr/loadgen-account-selection

Conversation

@SirTyson

@SirTyson SirTyson commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Update (2026-11-09)

Rebased onto the updated #5457 (the stack now includes #5453's merge of master @ 362dbc5). Four commits now sit on top of the original change.

  • Match externalized accounts in time proportional to the tx set. cleanupAccounts runs in valueExternalized on every node for every ledger. It used to allocate a set entry for the source of every transaction in the externalized set, then walk every in-use account with a map lookup. When ledgers fall behind the offered load, the in-use backlog grows, and that walk grew with it on the externalize path. In-use accounts are now indexed by public key, and each externalized transaction's source is looked up there. The cost no longer depends on the backlog, and the lookups allocate nothing. Which accounts are released, and when, is unchanged.
  • Stop re-registering loadgen metrics and copying envelopes per transaction. execute() used to resolve ten meters through the metrics registry for every transaction, and copied the envelope into a StellarMessage just to size it. It now uses the meters resolved at construction and sizes the message from the envelope. The recorded values are identical.
  • Hash externalized transactions without rebuilding their frames. To give the Rust overlay the full hashes of the externalized transactions, and loadgen their source accounts, processExternalized built a TransactionFrame for every transaction in the set on the main thread (about 7k per ledger at 3k TPS). That copied each envelope and built one operation frame per operation. The set's envelopes are now hashed in place (TxSetXDRFrame::getTransactionFullHashes), and loadgen reads sources straight from the envelopes. The overlay receives the same hashes in the same order, and loadgen releases the same accounts.
  • Tests. On this branch submission never rejects, so the recycling test's "no rejected transactions" assertion could not fail. It now relies on every transaction applying and on runs completing with synced sequence numbers. It also fails fast and runs in about a third of the time. New tests:
    • "synthetic SAC load recycles source accounts in overlay-only mode" runs perfnet's load (synthetic SAC payments, overlay-only, more transactions than accounts). It checks completion and the per-transaction meters, including loadgen.txn.bytes.
    • "externalized tx set hashes and sources match transaction frames" pins the hashes and sources from legacy and generalized sets (sequential and parallel phases; v0, v1, muxed and fee-bump envelopes) against frames built from the same set.

Known limit: in the MIXED_PREGEN_* modes, the Soroban stream fails the run when every source account has a transaction in flight, whereas the other modes wait. At perfnet's 33k accounts per generator, this needs a backlog of hundreds of slow ledgers, which is longer than a block_time_check run.

Testing:

  • New and changed tests by name: the six loadgen tests (including both new ones and the sped-up recycling test) pass.
  • [loadgen],[herder],[txset] (partitioned make check): 94/94 test cases pass.
  • Full suite at the final head e5ec7d4cb (which contains the whole stack): selftest 590 test cases / 2.85M assertions, check-sorobans, check-rust, check-nondet all pass.
  • An independent review found no correctness, determinism or safety issues in the original commit or the follow-ups.

The performance tables above are for the original commit on the previous base. Perfnet run of the final head (2026-11-09):

Perfnet, final head e5ec7d4cb (30 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered from all 30 nodes, 2 s target, 150 ms simulated apply, tx sets capped at 7,500; n = 1):

  • Correct: protocol 30 with millisecond close times on all nodes; no forks, all in sync, no invalid values/signatures, no restarts, no overlay errors during the load.
  • Verdict FAIL (window 1 p75 3.25 s on every node, p99 3.89 s), essentially the same speed as the previous image of this PR on the same fixture (p75 3.32 s, p99 3.92 s): about 3.1 s per ledger with full 7,500-tx ledgers and a growing backlog.
  • Per ledger: ~0.3 s externalize + close (incl. 150 ms apply), ~1.05 s tx-set build before nomination, ~1.34 s nomination (of which ~0.33 s tx-set fetch, ~0.34 s validation), ~0.28 s ballot. Every trigger was already due (backlog), so the leader's early tx-set preparation never ran and the build sits on the critical path. The new trigger metrics (Merge ms close time #5454) show lateness p75 0.25 ms and no prepare-start fallbacks.
  • This PR's changes: loadgen submit 7-10% cheaper (mean 2.71 vs 2.91 ms); demand timeouts -21% and duplicate flood bytes -21% (both only during the first minutes of load); neither is on the ledger critical path in this fixture.

Original description

The numbers below come from an earlier simulation environment and are kept for history. The stack is now measured on perfnet (30 validators in 4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, 900 s load); the run of this PR's final head will be posted here.

Description

This PR removes the linear account-selection operation from loadgen. Previously, each transaction selected a random position in an unordered set and walked to it. With large account pools, that traversal stalled submission callbacks on the main thread.

Available accounts now live in a vector. Selection uses a uniform random index, and removal swaps the selected account with the last entry before popping it. Both operations take constant time. Accounts return to the available pool after their ledger has applied; the test covers account recycling and consecutive loadgen runs.

Here, the control image is the PR this is based on, #5457. This is the 10 tier 1, 2 second target block latency test. Median block intervals are from the canonical node; p99 is the worst node over the full measurement window.

2,000 TPS

Metric Control Loadgen account selection
Median block interval 2.000 s 1.956 s
Worst-node p99 block interval 2.508 s 2.595 s
Mean loadgen submission batch 48.28 ms 2.68 ms
Harness verdict PASS PASS

3,000 TPS

Metric Control Loadgen account selection
Median block interval 2.363 s 2.341 s
Worst-node p99 block interval 4.524 s 3.105 s
Mean loadgen submission batch 88.31 ms 3.62 ms
Harness verdict FAIL FAIL

Submission overhead falls substantially at both loads. At 3,000 TPS, worst-node p99 improves, but both runs still fail the harness p75 threshold. Submission-batch means use timer count/sum differences across the ten load generators.

Checklist

  • Reviewed the contributing document.
  • One commit on top of Batch GETDATA transaction responses #5457, with no merge commits.
  • Formatting checked with make format.
  • Local build and Jenkins package build succeeded.
  • Relevant loadgen tests passed: 8 cases / 947 assertions, including real-apply account recycling and consecutive runs.
  • Performance comparison included above.

@SirTyson
SirTyson force-pushed the pr/loadgen-account-selection branch from 26e0f29 to 03a55f7 Compare September 16, 2026 02:56
@SirTyson
SirTyson force-pushed the pr/loadgen-account-selection branch from 03a55f7 to 195adf7 Compare September 16, 2026 03:23
@SirTyson
SirTyson force-pushed the pr/loadgen-account-selection branch from 195adf7 to e5ec7d4 Compare November 9, 2026 21:38
This was referenced Nov 9, 2026
@SirTyson

SirTyson commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Updated (force-pushed, 2026-11-09): rebased onto the updated #5457 (stack now includes the master merge), plus four follow-up commits: per-ledger account matching proportional to the tx set, no per-tx metric re-registration/envelope copies, hashing externalized txs without rebuilding frames, and test fixes. Full suite passes at this head; a perfnet run of it is in progress and will be posted here.

Commits on top of the base:

  • a49f7b22c Select loadgen source accounts in constant time
  • e94becc58 Match externalized loadgen accounts in time proportional to the tx set
  • ae3fce927 Stop re-registering loadgen metrics and copying envelopes per transaction
  • 458744ea1 Make the loadgen account recycling test check what it claims, faster
  • e5ec7d4cb Hash externalized transactions without rebuilding their frames

@SirTyson
SirTyson force-pushed the pr/loadgen-account-selection branch from e5ec7d4 to a3d0e18 Compare November 10, 2026 21:05
Port the focused account-pool change from 75f3fbc onto batched-getdata.
Use uniform indexed selection and swap-removal instead of walking the
available-account hash set for every transaction. Preserve release only
after application, including Soroban setup transitions.

Cover repeated account recycling and consecutive runs using real ledger
application, with more submissions than available accounts and no rejected
transactions.
cleanupAccounts runs on the main thread in valueExternalized for every
ledger, on every node, whether or not that node generates load. It hashed
and allocated a set entry for the source of every transaction in the
externalized set (about 6,000 to 7,500 per ledger in perfnet's 3,000 TPS
runs), then walked every in-use account with a std::map lookup. When
ledgers fall behind the offered load, the in-use backlog grows by tens of
accounts per ledger per generator, so the walk grew with it on the path
between externalize and the next ledger.

Keep in-use accounts indexed by public key instead. Accounts selected
since the previous call are indexed at the start of the next one (their
keys are known once TxGenerator has generated a transaction for them),
and each externalized transaction's source is looked up in that index. The
cost is now one allocation-free lookup per transaction in the set plus one
index insert per newly selected account, independent of the backlog, with an
early return when no account is waiting. Which accounts are released, and
when, is unchanged.

Add a test for perfnet's block_time_check load: synthetic SAC payments in
overlay-only mode with more transactions than source accounts, which only
completes if accounts return to the pool after their ledgers close.
…tion

LoadGenerator::execute built a fresh TxMetrics for every submitted
transaction, which resolved all ten loadgen meters through
MetricsRegistry::NewMeter: a MetricName allocation, a mutex, and two
searches of the registry's std::map each. It also copied the whole
envelope into a new StellarMessage (twice for classic payments) only to
measure its size.

Use the TxMetrics member resolved once at construction, which refers to
the same registry meters, and compute the message size as the 4-byte
message type discriminant plus the envelope's XDR size. The recorded
values are unchanged.

Extend the overlay-only synthetic SAC test to check the per-transaction
meters, including that loadgen.txn.bytes still equals the wire size of
each submitted StellarMessage.
The section asserted that no transaction was rejected, but on this branch
HerderImpl::recvTransaction hands every transaction to the overlay and
always returns TX_STATUS_PENDING, so loadgen.txn.rejected cannot move. The
checks that matter are that every transaction applies successfully and that
the run completes, which requires each account's cached sequence number to
match the ledger. Say so, drop the rejected assertion, and stop cranking as
soon as a run fails instead of waiting for the timeout.

Submit 2x the accounts at 200 tx/s instead of 3x at 100 tx/s. That still
forces every account through the pool more than once and stays within the
unique-account check (3 x 200 txs per one-second ledger = 600 <= 1,000
accounts), and each run takes about 10 s of submission instead of 30 s.
HerderImpl::processExternalized needs two things from every transaction
in the externalized set on the main thread: its full hash, so the Rust
overlay can drop it from the mempool, and (in BUILD_TESTS) its source
account, so loadgen can release it. It got both by calling
createTransactionFrames, which copies every envelope into a new
TransactionFrame (with an OperationFrame per operation) only to hash it
and read its source, then frees them all. That is thousands of frames per
ledger at perfnet's load.

Add TxSetXDRFrame::forEachTransactionEnvelope, which visits the set's
envelopes in createTransactionFrames' order, and getTransactionFullHashes,
which hashes each envelope's XDR in place: the same SHA-256 that
TransactionFrame::getFullHash and FeeBumpTransactionFrame::getFullHash
compute. LoadGenerator::cleanupAccounts takes the tx set and reads each
source from the envelope with getEnvelopeSourceID, which mirrors
getSourceID: the unmuxed source, or a fee bump's inner source. The overlay
receives the same hashes in the same order, and loadgen releases the same
accounts.

Add a test that builds legacy and generalized sets (sequential and
parallel phases; v0, v1, muxed and fee-bump envelopes) from XDR and checks
the visited envelopes, full hashes and sources against the frames
createTransactionFrames builds from the same set.
@SirTyson
SirTyson force-pushed the pr/loadgen-account-selection branch from a3d0e18 to 0fd7d0e Compare November 10, 2026 21:22
SirTyson added a commit that referenced this pull request Nov 10, 2026
## Update (2026-11-09)

This PR is rebased onto the reworked #5453, which merges current
`master` (362dbc5) into
`overlay-v2-shared`. That merge brings in the final #5423 (millisecond
close times, gated at protocol
30, with the protocol-30 bump). It also carries the overlay-v2
adaptations those need, including the
Rust overlay moving to `stellar-xdr` 30.0.0 to decode millisecond
`StellarValue`s.

**The old first commit ("Enable millisecond consensus close times at
protocol 28") is dropped.** The
master merge in #5453 supersedes it. It was an earlier draft of #5423
behind a compile-time
`MS_CLOSE_TIME` flag at protocol 28, with its own XDR pin and fixtures.
The overlay-v2-specific parts
needed for correctness are now in #5453.

This PR is now the trigger-timing work on top of millisecond close
times:

1. **Account for actual elapsed nomination time in the trigger drift
allowance** (ported unchanged
in substance). The next-ledger trigger is anchored on the previous
externalized close time C and
fires at `C + target`. It falls back to the local prepare-start anchor
when the time elapsed
since C exceeds the target plus an allowance for nomination. That
allowance used to be the sum of
nomination timeouts that had fired. Now it is the steady-clock time from
the local trigger to
entry into ballot. A slow round that never runs a timeout callback no
longer counts as clock
drift: slow tx-set fetch or validation, a busy main thread, or ballot
starting before the expired
timer runs. When the network is behind, the next trigger fires
immediately instead of waiting
out the nomination again. The extra wait a fallback can add is bounded
by our clock drift plus
   how late we triggered relative to the winning close time.
2. **Record the nomination trigger only when nomination proceeds**
(new). If a node skips
nomination because its clock is behind the last close time, it no longer
credits the wait for
   its peers to the allowance.
3. **Test the trigger anchor with sub-second close times** (new). With
close times 0, 250 and 999 ms
into the second, the next trigger is due exactly the target after the
close time, and the
   fallback boundary holds to the millisecond.
4. **Add trigger timing metrics** (new). Together they show whether a 2
s ledger is bound by apply,
   by nomination or by event-loop delay:
- `scp.timing.trigger-to-ballot` (timer): proposal construction plus
nomination, which is the
     drift allowance.
- `scp.trigger.lateness` (timer): how late the trigger callback ran
after its scheduled time.
- `scp.trigger.immediate` (meter): the next trigger was already due when
it was scheduled.

5. **Skip a delayed nomination emit that was already sent** (new,
test-only code path). With
`ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING`, a deferred nomination
emit could fire after
`processEnvelope` had already emitted the same statement. SCP then
rejected the duplicate and
threw "moved to a bad state (nomination)", which made the hidden
"consensus close time trigger
timer" test flaky under parallel load. The deferred emit now only sends
a statement newer than
   our last one. A new SCP test replays the race deterministically.

None of these commits change values, validation or ledger output. They
only change local trigger
scheduling, metrics, logs and BUILD_TESTS-only code.

**Follow-up under consideration:** for proposals rebuilt at the trigger,
propose the scheduled
trigger time as the close time (bounded by `now` and LCL+1 ms), so a
leader's trigger lateness no
longer shifts the network's cadence. We'll do it if
`scp.trigger.lateness` p75 exceeds about 20 ms
on perfnet.

Testing, at `285aa42fc92d` on top of #5453 `04fbddb7d`:
- New and changed tests: `[trigger-construction]` 8/8 (the
drift-allowance tests and the 3 new
  ones) and `[early-nomination]` 6/6 pass.
- `[herder]` (partitioned `make check`): 77/77 test cases pass,
including the hidden real-clock
  trigger-timer simulations.
- At `4f577bd25`: the new SCP test passes. A copy of the test without
the fix fails with the
original exception, so the test reproduces the race. "consensus close
time trigger timer" passes
3/3 on its own and 2/2 inside a partitioned `[herder],[scp]` `make
check` (97/97 test cases, full
  check passes both times).
- Final branch: restacked onto the final #5453 head `14de3972a` (two
test-only commits on top of `04fbddb7d`); the five patches are identical
(`git range-diff`). The full suite passes on the #5458 head that
contains this PR (selftest 590 test cases, check-sorobans, check-rust,
check-nondet).


---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet (30 validators in
4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated
apply, 900 s load); the runs of the final #5458 and #5459 heads contain
this PR and are reported on those PRs.

# Description

Merges in #5423.

This change adds ms closeTime. By itself, it doesn't improve TPS or
latency significantly. However, it reduces our long tail fo very slow
blocks, and also allows us to target intrasecond or subsecond block
times, which are not currently possible at whole second resolution.

In addition to taking the change from the PR, the 2nd commit makes some
minor adjustments to when we call the fallback timer. This was
intentionally conservative for main net launch assuming lots of drifting
clocks, but no reason we can't tighten it here.

Results, measured with 10 tier 1 topology (no watchers) with 2 second
block time target:

## 2000 TPS

| Metric | Original overlay-v2-shared |Changes |
|---|---:|---:|
| Median block interval | 2.090 s | 1.993 s |
| p99 block interval | 4.563 s |  4.298 s |
| Maximum block interval | 6.219 s | 5.090 s |
| Harness verdict | **FAIL** | **FAIL** |


Rebased on #5453 so opening
as a draft for now.

# Checklist
- [x] Reviewed the
[contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#opening-a-pull-request)
document
- [x] Rebased on top of master (no merge commits)
- [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio
extension)
- [x] Compiles
- [x] Ran all tests
- [ ] If change impacts performance, include supporting evidence per the
[performance
document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
SirTyson added a commit that referenced this pull request Nov 10, 2026
## Update (2026-11-09)

Rebased onto the updated #5454 (which now sits on #5453's merge of
`master` @ 362dbc5; the overlay crate moved to `stellar-xdr` 30.0.0
there). The original commit is unchanged apart from a test import
updated for stellar-xdr 30.0.0 (`stellar_xdr::curr::` to
`stellar_xdr::`). Eight commits on top, each with tests (an independent
review found no correctness, wire-format or deadlock issues in the
original commit; its low-severity findings are fixed below):

- **Stop serving a demand after a failed write.** A response job returns
at the first write that fails after the stream retries (usually a
disconnect), instead of looking up, writing and logging a failure for
every remaining batch. The requester re-demands unanswered hashes after
its response timeout.
- **Frame responses straight from the shared transaction bytes.** Each
frame's discriminant and envelope are copied directly into the batch,
with no per-transaction allocation; a transaction larger than the batch
budget is written as two borrowed parts with no copy. Each batch is
looked up under one TX-buffer lock, released before frames are copied or
written, so local submissions on the event loop never wait for a
response copy.
- **Buffer inbound TX stream reads (64 KiB).** One QUIC read takes in
whatever has arrived, often a whole response batch, instead of two reads
per frame. The reader never waits for more data than is available.
- **Queue INVs for all peers under one lock**, for both relayed and
locally submitted transactions, instead of one batcher lock per peer.
- **Handle each received INV batch with one seen-set lock and one
tracker lock**, instead of both locks per advertised hash. Request
decisions and their order are unchanged.
- **More tests:** batching edges (an oversized transaction first, two in
a row, frames exactly at the budget, carried hits, misses); a failed
batch write retried whole on a reopened stream; buffered reads; delivery
of INV batches completed by either path; INV request decisions and
order.

Wire bytes are unchanged. Testing on the rebased branch:
- 194 overlay crate tests pass (`cargo test` in `overlay/`, 3 ignored);
`cargo fmt --check` is clean.
- The standard build succeeds, and 25 C++ overlay-IPC test cases (136
assertions) pass: `[overlay-ipc]`, `[overlay][ipc]` and the 10-node
`[overlay-ipc-large]` consensus case.
- The full suite passes on the #5458 head that contains this PR
(selftest 590 test cases, check-sorobans, check-rust with these overlay
tests, check-nondet).

The follow-up commits have not been measured separately on perfnet; the
numbers above are for the original commit on the previous base. The
perfnet run of the final #5458 head (which contains this PR) is reported
on #5458.

**Next lever (not in this PR):** a node waits up to 100 ms (checked
every 50 ms) before announcing new transactions to a peer, so each
flooding hop adds 100-150 ms before the peer can request them. A perfnet
A/B of shorter delays (50 ms and 25 ms) is the planned follow-up,
comparing tx pull latency, tx-set fetch p99 and block intervals against
bandwidth and CPU.

---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet (30 validators in
4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated
apply, 900 s load); the runs of the final #5458 and #5459 heads contain
this PR and are reported on those PRs.

# Description

This PR batches GETDATA transaction responses in the Rust overlay.
Previously, a request for many transactions created a separate task and
stream write for every response. Under load, that produced unnecessary
scheduling and stream-lock overhead, and a slow peer could leave the
entire response demand queued in memory.

We now serve each request in one asynchronous job, combining complete
transaction frames into writes of up to 64 KiB. Each transaction keeps
its existing wire framing, and the writes use the existing stream
priorities and retry policy. Transactions larger than the batch budget
are sent individually.

Here, the control image is the PR this is based on
#5454. This is the 10 tier
1, 2 second target block latency test. Median block intervals are from
the canonical node; p99 is the worst node over the full measurement
window.

## 2,000 TPS

| Metric | Control | Batched GETDATA |
|---|---:|---:|
| Median block interval | 1.993 s | 2.000 s |
| Worst-node p99 block interval | 5.090 s | 2.508 s |
| Tx-set fetch p99 | 1.720 s | 0.778 s |
| Harness verdict | FAIL | PASS |

## 3,000 TPS

| Metric | Control  | Batched GETDATA|
|---|---:|---:|
| Median block interval | 2.434 s | 2.363 s |
| Worst-node p99 block interval | 4.122 s | 4.524 s |
| Tx-set fetch p99 | 1.757 s | 1.087 s |
| Harness verdict | FAIL | FAIL |

At 2,000 TPS, the largest improvement is in the tail: worst-node p99
falls by about half, and nearly all tx-set fetches longer than two
seconds disappear. This is consistent with reducing transaction-response
write overhead and contention affecting consensus transfers.

# Checklist

- [x] Reviewed the contributing document.
- [x] One commit on top of #5454, with no merge commits. (Now nine: the
original plus eight follow-ups, see the update above.)
- [x] Formatting checked with `make format` and `cargo fmt --check`.
- [x] Release overlay build and Jenkins package build succeeded.
- [x] Relevant tests passed: 185 Rust tests (3 ignored), plus 17 C++
overlay-IPC cases / 82 assertions.
- [x] Performance comparison included above.
Base automatically changed from pr/batched-getdata to pr/merge-ms-close-time-base November 10, 2026 22:02
@SirTyson
SirTyson marked this pull request as ready for review November 10, 2026 22:04
@SirTyson
SirTyson merged commit f5353da into pr/merge-ms-close-time-base Nov 10, 2026
3 of 5 checks passed
@SirTyson
SirTyson deleted the pr/loadgen-account-selection branch November 10, 2026 22:04
SirTyson added a commit that referenced this pull request Nov 10, 2026
## Update (2026-11-10): cheaper leader build, ballot-start signature
prewarm, IPC size fix; perfnet passes on c6id

Head is now `ebdd73635`, 8 commits on top of `c26b191fa`. They fix the
two things the 2026-11-09 perfnet run found, plus a liveness bug the
next run found.

### Leader build
On perfnet the leader's 1.1 s build was on the critical path every
ledger.
- `ba4955347` **validates only what selection can use.** The leader
still pulls 2x the capacity, but validates each phase in mempool order
only until it holds 5/4 of the capacity in valid transactions. If some
are invalid it refills: twice the shortfall, then the rest. The selected
set is the same unless selection would have reached past the validated
prefix.
- `a1bffae42` hashes the candidates on the validation workers, not the
main thread. That was 18% of a cold build's instructions on hosts
without SHA extensions, such as c5d.
- `966d3bb41` precomputes the parallel builder's sort keys.
- `95be616c7` decodes mempool pulls straight from the IPC payload,
without copying.
- `2a30c1dc0` adds phase timers
`herder.txset.build-{pull,frames,trim,select,xdr,handoff,roundtrip,final-check}`
and a hidden leader-build benchmark.
- Laptop benchmark (15,000 SAC candidates, cap 7,500, 4 tasks, n=5
medians): cold build 261 -> 168 ms (-36%), warm 124 -> 84 ms (-32%).
Main-thread instructions -41% (callgrind).

### Signature prewarm from ballot start, on every validator
(`a91547bec`)
The 2026-11-09 follower prewarm covered only about half of the leader's
set. The fees are equal, so the mempool orders transactions by hash, but
selection breaks ties randomly. The leader also had no prewarm.
- Now, when slot N starts balloting, every validator verifies the
signatures of exactly the window that the N+1 build validates first.
This runs while N finishes balloting, externalizes and closes. The main
thread only makes the overlay round trip; decoding and verification run
on background tasks.
- New metrics: `herder.prewarm.{pull,duration}`,
`herder.prewarm.{txs,verified,incomplete}` and
`herder.txset.{build,validate}-sig-cache-{hit,miss}`.
- Laptop: a build after a prewarm takes 83 ms (cold 169). Received-set
validation takes 38 ms (cold 102). 0 cache misses.
- `0cf227b94`: Core's tx-set fetch timer is now `scp.fetch.txset`, with
`scp.fetch.txset-retry`. `overlay.fetch.txset` is now only the Rust
overlay's time; it had been fed by both.

### Bug fix: oversized IPC payloads (`ebdd73635`)
Perfnet found this on `a91547bec`, on c5d.
- **What happened:** 80 s into the load the whole network stopped at one
ledger. 16 of 30 nodes logged `Overlay IPC connection closed` within
0.35 s of each other. The rest then lost quorum, and all lost sync.
- **Cause:** the prewarm pulls 2 x 7,600 + 7,500 transactions (22,700
SAC payments, about 17.2 MB) once the mempool backs up. Both ends of the
Core-overlay IPC rejected payloads over 16 MiB, and Core's reader treats
a rejected length as a corrupt stream: it stops reading and never
recovers. Every validator starts the pull at the same ballot.
- **Fix:**
- The limit is now 64 MiB on both sides (`ipc::MAX_PAYLOAD_BYTES` and
`MAX_PAYLOAD_SIZE`, which must match).
- The overlay returns the longest prefix of the top transactions that
fits, with a warning, so no mempool can produce an oversized response.
- Neither side sends an over-limit payload: Rust refuses with
`PayloadTooLarge`, and Core refuses before writing anything.
  - Core now logs the type and length when it rejects one.
- This also covers a maximum-size peer tx set (`MAX_TXSET_XDR_SIZE` + 32
bytes), which could have hit the old limit too.
- **Test:** a new `[overlay-ipc]` test with the real Rust overlay. A 20
MiB top-transactions response arrives whole; with the old code the pull
returns nothing (`0 == 20`). An 80 MiB pull is truncated to the 63
highest-fee transactions, and the IPC keeps working.

### Testing
- **`a91547bec`:**
  - Laptop: new and changed tests, and the step-2 tags (149 test cases).
- Devbox: full suite `--all-versions` PASS (selftest 604 test cases /
8,638,027 assertions, check-nondet, check-rust).
- **`ebdd73635`** (devbox):
- New tests: the `[overlay-ipc]` test above, plus IPC tests for a 20 MB
message, an over-limit length (rejected, with a receive timeout so a
regression fails instead of hanging) and an over-limit send (nothing
written). Rust tests cover truncation and the send guard.
- `[ipc]` (7 test cases), `[overlay-ipc]` (20), partitioned
`[overlay]~[ipc-crossproc]` and the Rust `ipc::` tests (23): all pass.
- Two pre-existing failures reproduce on `a91547bec` without the change:
- The Rust `test_tx_not_blocked_by_scp_flood` fails its own validity
check (an SCP flood of 6.7 ms is under its 10 ms floor).
- The hidden `IPC cross-process communication` test needs an unbuilt
`overlay-stub` binary.
- Full suite `--all-versions` on `ebdd73635` (devbox): PASS. selftest
605 test cases / 8,638,079 assertions across protocol versions,
check-nondet, libsodium (77) and xdrpp (7), check-rust and the soroban
host tests (103 `test result: ok`, 0 failed). The overlay crate's own
tests run separately; see above.

### Perfnet, head `ebdd73635`
30 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered, 2
s target, 150 ms simulated apply, tx sets capped at 7,500, 900 s
measured. One run per instance type, both on the same day: c5d.2xlarge
(perfnet's default, Cascade Lake, no SHA-NI) and c6id.2xlarge (Ice Lake,
SHA-NI). Same 8 vCPU / 16 GiB.

**c6id.2xlarge: PASS.** All four windows on target, inclusion passed
(2,880,000 txs), all checks passed.

| Window | avg p75 | avg p99 | p75 range over nodes |
|---|---:|---:|---|
| 1 | 2,112 ms | 2,334 ms | 2,103-2,129 ms |
| 2 | 2,099 ms | 2,391 ms | 2,091-2,113 ms |
| 3 | 2,101 ms | 2,324 ms | 2,092-2,110 ms |
| 4 | 2,099 ms | 2,354 ms | 2,089-2,110 ms |

**c5d.2xlarge: FAIL** in window 1, but 19% faster than the previous head
and #5458. The c5d network fell behind the offered load, so sets were
full at 7,500 and the mempool backed up. That's exactly the case that
stalled `a91547bec`, and here there were no IPC disconnects or sync
losses. Correctness passed: error counters, in sync, no fork, still
closing. Inclusion was skipped because the load stopped after window 1.

| c5d, window 1 (5-min means) | #5458 head | #5459 `c26b191fa` | **#5459
`ebdd73635`** |
|---|---:|---:|---:|
| avg p75 / p99 | 3,250 / 3,891 ms | 3,264 / 3,672 ms | **2,637 / 2,920
ms** |
| Ledgers per 5 min | 96 | 96 | 119 |
| Trigger to first ballot | 2.52 s | 2.55 s | 1.94 s |
| Leader build | 1.15 s | 1.11 s | 0.68 s (candidate validation 0.20 s)
|
| Received-set validation | 0.34 s | 0.52 s | 0.33 s |
| Nomination timeouts per node | 5 | 106 | 34 |
| Ledger close | 0.229 s | 0.228 s | 0.229 s |

Both instance types, same build:

| | c5d | c6id |
|---|---:|---:|
| Ledger interval | about 2.53 s | about 2.03 s |
| Txs per ledger | 7,500 (backlog) | about 6,100 (kept up with 3,000
TPS) |
| Trigger to first ballot | 1.94 s | 1.22 s |
| Leader build | 0.68 s | 0.27 s |
| Received-set validation | 0.33 s | 0.20 s |
| Signature-cache coverage: build / validation | 89.5% / 98.7% | 87.9% /
97.8% |
| Prewarm: mean duration / main-thread pull / stopped early | 0.21 s /
22 ms / 0.1% | 0.10 s / 7 ms / 0% |
| Ledger close | 0.229 s | 0.164 s |

Notes:
- **No A/B on c6id.** #5458 has not been run on c6id, so the c6id PASS
shows this head can hold the target on current-generation x86. It
doesn't show how much of that comes from this PR. Every earlier run of
this fixture was on c5d.
- **Two regimes.** On c6id the network keeps up, so sets stay at about
6,100 transactions and triggers aren't all due at once; #5454's early
preparation runs, giving about 2 builds per ledger network-wide. On c5d
it falls behind, sets are full, and every ledger is heavier. So
per-ledger timings aren't like for like across the two.
- **What c5d still needs** (about 0.24 s to pass):
- Received-set validation still takes 0.33 s with 98.7% of its
signatures cached, so the rest of validation is now the cost.
  - About 245 ms of the c5d build is outside the phase timers.
- The remaining trigger-to-ballot time beyond build, fetch and
validation is about 0.57 s of nomination messaging across 4 regions.
- n = 1 per instance type.

---

## Previous update (2026-11-09)

_Genesis-upgrade fix, fetch robustness, follower signature prewarm._

Rebased onto the updated stack (#5453 now merges `master` @ 362dbc5:
protocol 30, millisecond close times; #5454, #5457 and #5458 updated the
same day). The PR's original four commits are kept (adapted to the new
base: stellar-xdr 30.0.0 paths, #5457's batched frame writes); ten
commits are added on top.

### Bug fix: networks starting at the genesis protocol stalled
(465f6a4)

**Root cause.** With leader-only construction (5fff543), followers no
longer build a tx set at
the trigger and must fetch the leader's. Before protocol 20 a ledger's
set is a legacy
`TransactionSet`, and the Rust overlay only carries
`GeneralizedTransactionSet`s (Core caches,
Rust decodes and Core receives only that type). So a follower could get
the leader's set only by
being promoted to leader by nomination timeouts and building it itself,
one slow round at a time.
Before this PR every validator built the identical empty legacy set (it
depends only on the
previous ledger; upgrades live in the `StellarValue`), so it was always
local. On perfnet (30 nodes)
22 nodes reached ledger 2 and stalled, 8 stayed at ledger 1. The upgrade
ledger itself
(LCL < 20) has the same problem; ledgers from protocol 20 on are
unaffected.

**Fix.** When the LCL is before protocol 20, every validator constructs
its nomination value at
the trigger again, as before; it is still voted only by round leaders.
Leader-only construction
is unchanged from protocol 20 on.

**Test.** `Rust overlay upgrades from genesis through legacy tx sets`: 4
nodes, genesis protocol 0,
four legacy ledgers at normal cadence, an upgrade straight to protocol
30, three more ledgers with
real (generalized) tx sets. It fails without the fix (a node stuck at
ledger 2) and passes with it.
Ablation on the final stack: disabling only this fix, with all the other
changes below in place,
still fails (2/2), so they don't mask the bug.

### Robustness on the same path
- 193fee3: a locally built set completes a pending fetch of the same
hash, so envelopes waiting
for it are processed at once (the IPC path had lost upstream's
ItemFetcher behaviour).
- 942a6b3: a follower promoted to leader after losing sync can still
propose; tracking is
checked at the trigger, as before. This is what made the perfnet stall
permanent.
- ab4eb14, 5964ce6: unanswered tx-set requests are retried and
reported. Core re-requests
undecided slots' sets with backoff (1, 2, 4, then every 8 s), warns on
every doubling of the
request count and counts `overlay.fetch.txset-retry`; Rust sends a retry
to a random other peer
and prunes stale reservations by slot. Core also checks a delivered
set's content hash.
- 5c159ae: test for a leader whose own build completes a pending fetch
and externalizes the
  slot inside SCP's nomination supplier.
- Rebase onto the new base: the test-only nomination override from #5481
now also hands its
substituted set to the overlay (31e6147), since this PR caches the
built set earlier.

### Performance
- 3903e4e, f2643ba: follower signature prewarm. The received-set
validation timer rose (about
158 -> 205 ms) because followers no longer validated their own mempool,
which used to warm the
signature-verification cache. Benchmark (6,000 one-signature txs, 4
tasks): 67 ms cold vs
10.5 ms warm. Followers now pull their top mempool transactions once at
the trigger and verify
master-key signatures on background tasks (at most 20 verifications per
tx; two worker threads
left free; skipped if the node already voted; stopped when validation, a
local build or
  externalization starts). They still build and compress nothing.
- c26b191: no early tx-set preparation when the trigger is already
due. Both timers fired at the
same instant in unspecified order, so a predicted leader could build its
set twice.
- 2ac0af9: no deep copies of tx-set XDR on the overlay IPC paths
(leader serializes in place,
follower decodes in place and moves into the frame). Sub-millisecond for
classic sets.

### Testing
- Laptop, on c26b191: new and changed tests (24, including the genesis
repro and the trigger,
  early-nomination and #5481 tests) and partitioned `make check` for

`[herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[nominationprotocol]~[.]`
  (143 test cases) plus the overlay's Rust tests (169 + 32 + 3) pass.
- Full suite `--all-versions` (devbox, at c26b191): selftest 598 test
cases / 8.5M assertions across protocol versions, check-sorobans,
check-rust and check-nondet pass.
- **Perfnet, head `c26b191fa`** (30 validators in 4 AWS regions, 10
tier-1 orgs, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply,
tx sets capped at 7,500; n = 1; run in parallel with the #5458 head on
the same fixture):
- **The genesis-upgrade bug is fixed on a real network:** all 30 nodes
synced in 43 s from genesis through the legacy ledgers to protocol 30
(the previous image of this PR left 22 nodes at ledger 2 and 8 at ledger
1).
- Correct under load: no forks, all in sync, no invalid
values/signatures, no restarts, no `NOT IN CACHE`/parse/cache-drop
errors; 11 tx-set fetch retries, all resolved on the first re-request.
- Leader-only construction works as designed (~1 build per ledger
network-wide instead of 30), and zstd shrinks each 7,500-tx SAC set from
~5.6 MB to ~1.0 MB (5.7x, ~7.5 ms to encode).
- Verdict FAIL at the same speed as the #5458 head (window-1 p75 3.26 s
vs 3.25 s): the leader's ~1.1 s build stays on the critical path, and
follower validation of the received set was 0.52 s vs 0.34 s on #5458
(whose followers had warmed their signature cache by building), so the
follower prewarm did not recover it at this load. Nomination round 1 (2
s timeout) times out on almost every ledger.
- Follow-up: done in the 2026-11-10 update above (leader build,
ballot-start prewarm, and the IPC size fix found by the next run).

---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet; the run of this
PR's final head will be posted in the update above.

# Description

This PR compresses transaction-set responses in the Rust overlay and
shares the encoded bytes across peer sends. Every response uses the
mandatory zstd protocol; there is no legacy/raw fallback or optional
lazy encoding state. The leader hands the final XDR to Rust before
Core's final validation, so compression can overlap that validation.
Receivers retain the original compressed bytes, decompress and verify
the set, then reuse the retained encoding when forwarding it.

A separate commit, `0ed3e540`, removes private proposals on non-leader
nodes. Followers enter nomination with a lazy value supplier and follow
received proposals without pulling the mempool or constructing a set.
First-round leaders can prepare early; a promoted leader constructs a
value only when it needs one. The driver caches that value across
retries, and externalization stops nomination even when no local value
was built. Tests cover timeout promotion, current close times, following
another leader, unavailable values, and reentrant nomination stop.

This PR builds on #5458. The
comparison below evaluates these revisions against the **original
compressed implementation of this PR**, using the same 10-tier-1,
30-validator fixture: 3,000 offered TPS, 6,000 transactions/block,
2-second target, 8 CPUs/16 GiB per validator, 60-second warmup and
900-second measurement, with simulated apply.

## 3,000 offered TPS

| Metric | Original compression | Updated leader-only path |
|---|---:|---:|
| Median block interval | 2.018 s | 2.120 s |
| Worst-node full-window p99 block interval | 3.834 s | 3.836 s |
| Tx-set fetch p99 | 0.617 s | 0.601 s |
| Completed TPS | 2,926.67 | 2,806.67 |
| Harness verdict | PASS | FAIL (ob-0 only) |

Median intervals and completed throughput use the canonical node;
worst-node p99 uses the fixed measurement window. The harness uses a
different histogram/window: `ob-0` recorded 4.989 s against its 4-second
limit, while the other 29 nodes and inclusion passed. Completed
throughput fell 4.1% and median block interval rose 5.1%; this run does
**not** establish that the revised path matches or improves the original
compression baseline. These are single cloud runs with normal host
placement.

Leader-only construction is verified: 886 builds/compressions across 421
closed ledgers, all correlated with asserted leadership, with no
follower build or received-set recompression detected. Multiple
nomination leaders can build in a ledger. Compared with the intermediate
eager implementation, local compressions fell from 13,019 to 893 in
their respective 900-second windows, roughly a 93% reduction per ledger.
Payload savings remain 80.0%. The sampled received-set validation timer
rose from about 158 ms to 205 ms versus that intermediate build; the
precise cause is not isolated by this run.

# Checklist

- [x] Reviewed the contributing document.
- [x] Kept leader-only proposal construction in its own commit on this
PR's branch.
- [x] Formatting checked with `make format` and `cargo fmt --check`.
- [x] Local build, Jenkins package/image build, and exact-head Clang/GCC
CI passed.
- [x] Rust suite: 194 passed, 3 ignored; the leader-only follow-up
leaves those validated Rust sources unchanged.
- [x] Core validation: 41 focused SCP/nomination/tx-set cases, 17
overlay IPC cases, and all 72 Herder cases covered. PostgreSQL-dependent
cases were rerun after configuring the local database fixture.
- [x] Performance comparison, runtime identity, resource checks and
sealed-artifact audit completed.
- [ ] Performance matches or improves the original compressed baseline.
SirTyson added a commit that referenced this pull request Nov 10, 2026
…der schedule (#5496)

Brings the stacked PRs that were merged into
`pr/merge-ms-close-time-base` into `overlay-v2-shared`:

- #5457 Batch GETDATA transaction responses
- #5458 Select loadgen source accounts in constant time
- #5459 Compress and share transaction set responses
- #5495 Prototype: replace SCP nomination with a deterministic leader
schedule
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant