Skip to content

Merge ms close time - #5454

Merged
SirTyson merged 173 commits into
stellar:overlay-v2-sharedfrom
SirTyson:merge-ms-close-time
Nov 10, 2026
Merged

SirTyson merged 173 commits into
stellar:overlay-v2-sharedfrom
SirTyson:merge-ms-close-time

Conversation

@SirTyson

@SirTyson SirTyson commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Update (2026-11-09)

This PR is rebased onto the reworked #5453, which merges current master (362dbc5) into
overlay-v2-shared. That merge brings in the final #5423 (millisecond close times, gated at protocol
30, with the protocol-30 bump). It also carries the overlay-v2 adaptations those need, including the
Rust overlay moving to stellar-xdr 30.0.0 to decode millisecond StellarValues.

The old first commit ("Enable millisecond consensus close times at protocol 28") is dropped. The
master merge in #5453 supersedes it. It was an earlier draft of #5423 behind a compile-time
MS_CLOSE_TIME flag at protocol 28, with its own XDR pin and fixtures. The overlay-v2-specific parts
needed for correctness are now in #5453.

This PR is now the trigger-timing work on top of millisecond close times:

  1. Account for actual elapsed nomination time in the trigger drift allowance (ported unchanged
    in substance). The next-ledger trigger is anchored on the previous externalized close time C and
    fires at C + target. It falls back to the local prepare-start anchor when the time elapsed
    since C exceeds the target plus an allowance for nomination. That allowance used to be the sum of
    nomination timeouts that had fired. Now it is the steady-clock time from the local trigger to
    entry into ballot. A slow round that never runs a timeout callback no longer counts as clock
    drift: slow tx-set fetch or validation, a busy main thread, or ballot starting before the expired
    timer runs. When the network is behind, the next trigger fires immediately instead of waiting
    out the nomination again. The extra wait a fallback can add is bounded by our clock drift plus
    how late we triggered relative to the winning close time.

  2. Record the nomination trigger only when nomination proceeds (new). If a node skips
    nomination because its clock is behind the last close time, it no longer credits the wait for
    its peers to the allowance.

  3. Test the trigger anchor with sub-second close times (new). With close times 0, 250 and 999 ms
    into the second, the next trigger is due exactly the target after the close time, and the
    fallback boundary holds to the millisecond.

  4. Add trigger timing metrics (new). Together they show whether a 2 s ledger is bound by apply,
    by nomination or by event-loop delay:

    • scp.timing.trigger-to-ballot (timer): proposal construction plus nomination, which is the
      drift allowance.
    • scp.trigger.lateness (timer): how late the trigger callback ran after its scheduled time.
    • scp.trigger.immediate (meter): the next trigger was already due when it was scheduled.
  5. Skip a delayed nomination emit that was already sent (new, test-only code path). With
    ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING, a deferred nomination emit could fire after
    processEnvelope had already emitted the same statement. SCP then rejected the duplicate and
    threw "moved to a bad state (nomination)", which made the hidden "consensus close time trigger
    timer" test flaky under parallel load. The deferred emit now only sends a statement newer than
    our last one. A new SCP test replays the race deterministically.

None of these commits change values, validation or ledger output. They only change local trigger
scheduling, metrics, logs and BUILD_TESTS-only code.

Follow-up under consideration: for proposals rebuilt at the trigger, propose the scheduled
trigger time as the close time (bounded by now and LCL+1 ms), so a leader's trigger lateness no
longer shifts the network's cadence. We'll do it if scp.trigger.lateness p75 exceeds about 20 ms
on perfnet.

Testing, at 285aa42fc92d on top of #5453 04fbddb7d:

  • New and changed tests: [trigger-construction] 8/8 (the drift-allowance tests and the 3 new
    ones) and [early-nomination] 6/6 pass.
  • [herder] (partitioned make check): 77/77 test cases pass, including the hidden real-clock
    trigger-timer simulations.
  • At 4f577bd25: the new SCP test passes. A copy of the test without the fix fails with the
    original exception, so the test reproduces the race. "consensus close time trigger timer" passes
    3/3 on its own and 2/2 inside a partitioned [herder],[scp] make check (97/97 test cases, full
    check passes both times).
  • Final branch: restacked onto the final Merge parallel txset changes #5453 head 14de3972a (two test-only commits on top of 04fbddb7d); the five patches are identical (git range-diff). The full suite passes on the Select loadgen source accounts in constant time #5458 head that contains this PR (selftest 590 test cases, check-sorobans, check-rust, check-nondet).

Original description

The numbers below come from an earlier simulation environment and are kept for history. The stack is now measured on perfnet (30 validators in 4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, 900 s load); the runs of the final #5458 and #5459 heads contain this PR and are reported on those PRs.

Description

Merges in #5423.

This change adds ms closeTime. By itself, it doesn't improve TPS or latency significantly. However, it reduces our long tail fo very slow blocks, and also allows us to target intrasecond or subsecond block times, which are not currently possible at whole second resolution.

In addition to taking the change from the PR, the 2nd commit makes some minor adjustments to when we call the fallback timer. This was intentionally conservative for main net launch assuming lots of drifting clocks, but no reason we can't tighten it here.

Results, measured with 10 tier 1 topology (no watchers) with 2 second block time target:

2000 TPS

Metric Original overlay-v2-shared Changes
Median block interval 2.090 s 1.993 s
p99 block interval 4.563 s 4.298 s
Maximum block interval 6.219 s 5.090 s
Harness verdict FAIL FAIL

Rebased on #5453 so opening as a draft for now.

Checklist

  • Reviewed the contributing document
  • Rebased on top of master (no merge commits)
  • Ran clang-format v8.0.0 (via make format or the Visual Studio extension)
  • Compiles
  • Ran all tests
  • If change impacts performance, include supporting evidence per the performance document

kanwalpreetd and others added 30 commits June 29, 2026 11:46
…#5340)

Adds a non-blocking CI job to run Horizon integration tests on pull
requests and merges to master by compiling Core from source.
just a little more peer performance tuning.
Resolves stellar#5363

Test had a race between network externalizing new ledgers and new node
joining with a gap larger than default MAX_SLOTS_TO_REMEMBER=12.
Solution is to bump the config.
# Description

This adds a new mode to apply load, allowing us for more "end to end"
simulations. In the new mode, we time and simulate ballot phase through
the end of apply, but in a single node network such that SCP state
transitions occur instantly. This is useful, as it allows us to
benchmark the processing overhead associated with consensus, as well as
measure cache performance and dedup similar work done during tx set
validation and the later application of this transaction set.

The motivation for this test was from my recent block latency
experiments. I noticed how optimizations in network calls, reducing
bandwidth, etc were not moving the needle on actual block latency. It
turned out that consensus latency was greatly affected by CPU based
bottlenecks, not just network calls. On supercluster, it's challenging
to benchmark CPU costs of concensus, as you can't easily run tracy or
profilers and have limited visibility to single node performance. This
test allows us to much more easily measure and improve the processing
heavy aspects of concensus. In overlay-v2-shared for a block with 6K SAC
transfers, we saw about 300 ms of non-network "ingestion" time when
receiving a tx set and 500 ms of apply. Imo this was a significant blind
spot in our single node apply load tests previously.

Note that the actual number produced by this test is meaningless, as
consensus without a network tells us very little about overall
performance. However, the reported phase timings are very useful in
identifying bottlenecks and comparing solutions. This simulates the
processing done by a non-leader node. In the future, it may also be
helpful to add a mode for block construction as well.

# Checklist
- [x] Reviewed the
[contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes)
document
- [x] Rebased on top of master (no merge commits)
- [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio
extension)
- [x] Compiles
- [x] Ran all tests
- [ ] If change impacts performance, include supporting evidence per the
[performance
document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
The state data structures are rather large and nested (due to containing XDR), so when they get large de-allocation may take up to a few ms. We don't really need to block the apply thread on this de-allocation, so we can hide it in the background thread.
Currently `getAllEntries` call seals the LTX and unnecessarily copies every entry, which adds up to a few ms if thousands of entries are involved.

With this change we just return the references to all the entries. Since this call is used just once in the 'production' path, and the results are only used in a couple of read-only calls, it shouldn't be hard to maintain the lifetime correctness for this method.
Resolves stellar#704.

This PR clamps the results of `ApplicableTxSetFrame::getTotalFees` and
`ApplicableTxSetGrame::getTotalInclusionFees` to `INT64_MAX` so that
these functions do not overflow. The only caller of these functions is
`compareTxSets`, and the only caller of that function is
`HerderSCPDriver::combineCandidates`. I think this is why stellar#704 ascribes
the bug to `combineCandidates`, but I'd argue it's lower down in these
fee summing functions.

I've verified that this fixes the PoC in stellar#704. The unit test I added is
much more targeted; it focuses on the underlying functions themselves.
Skip background tx sig verification for unauthenticated peers
Clamp additions in tx set fee summing functions
- Make the Node 24 env-var comment date-agnostic (2026-06-16 has passed)
- Remove the unused Corepack step; this repo has no package.json, so it
  only ever shimmed yarn/pnpm that are never invoked
- Replace dtolnay/rust-toolchain with `rustup update`, matching the
  convention in this repo's other Rust workflows and dropping a
  third-party action from a security workflow

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
graydon and others added 10 commits November 2, 2026 15:54
Just adds a metric that monitors the size of the tcmalloc heap
Derive the protocol from MAX_SUPPORTED_PROTOCOL_VERSION + 1 in src/util/ProtocolVersion.h instead of hardcoding it, since the test build enables next. Skip Horizon's stellar-rpc tests, as no released RPC image supports the next protocol.
…ellar#5477)

The RPC and Horizon integration tests build core with next enabled, so
derive their protocol from `MAX_SUPPORTED_PROTOCOL_VERSION + 1` instead
of hardcoding it (the hardcoded RPC `'27'` is failing). Horizon's
stellar-rpc tests are skipped since no released RPC image supports the
next protocol; Horizon caps the tested protocol at its own max.
Brings overlay-v2-shared up to date with master. The previous master
merge (b90d48e, stellar#5428) was recorded with master's tip 100cc38 missing
from its parents, so this merge used 100cc38 as its base (temporary
graft of b90d48e, removed afterwards).

Notable master changes brought in:
- Protocol 30: CURRENT_LEDGER_PROTOCOL_VERSION follows
  MAX_SUPPORTED_PROTOCOL_VERSION; CAP-0084 and CAP-0087 ungated, p29 and
  p30 Soroban hosts (stellar#5472, stellar#5478, stellar#5484, stellar#5483).
- Millisecond-resolution close times at protocol 30 (stellar#5423):
  ConsensusTime/ApplyTime types, STELLAR_VALUE_SIGNED_MS and
  STELLAR_VALUE_EMPTY_TX_SET_MS.
- Tx set validity hardening (stellar#5481), lower max message size in protocol
  29 (stellar#733), sub-second ledger edge-case fixes.
- Simplified ledger view hierarchy (stellar#5461); the overlay-only-mode
  sequence number skip now lives in TransactionFrame::checkValid.
- Banned accounts deprecated (CAP-0077 is active).
- Apply-path performance work (stellar#5402, stellar#5411, stellar#5414, stellar#5415, stellar#5418, stellar#5421,
  stellar#5436, stellar#5286), apply-load tx set construction mode (stellar#5448), --check-lcm
  (stellar#5435), v29.0.0 release merge.

Conflict resolution:
- Files overlay-v2-shared deleted and master modified (the C++ overlay,
  TransactionQueue, TransactionQueueTests, UpgradesTests, private/Horizon/RPC
  workflows): kept deleted. Master's changes to them are C++ overlay
  peer/flow-control logic, tx queue admission and tests for those.
- Herder: kept the Rust-overlay submission path (recvTransaction returns
  TxSubmitStatus) and dropped the deprecated `force` parameter as master did.
  triggerNextLedger keeps the prepared/built tx set path from the Rust
  mempool, now with ConsensusTime close times and master's apply-time
  close-time offset. The early tx set preparation path (prepareTxSet,
  buildTxSet, setupTriggerNextLedger) takes ConsensusTime. Kept master's
  mNominatedTxSetOverride test hook (applied before the tx set is cached in
  the Rust overlay) and the herder.txset.build timer (now timing
  buildTxSet). Dropped the banned-accounts setup and tx queue maintenance.
- AppConnector: dropped both ov2's maybeUpdateImmutableLedgerView and
  master's C++-overlay getOverlayThreadSnapshot.
- LoadGenerator: master's CachedTestAccountPtr with ov2's TxSubmitStatus.
- Tests: kept ov2's removal of tests that need the C++ overlay, standalone
  mode or the tx queue (standalone, manualclose, flooding, tx queue limits,
  far-future slot cleanup, herderExternalizesValuesWithProtocol, MAX_SAC_TPS,
  custom token, soroswap and noisy binary search apply-load tests). Added
  master's ms close time HerderTests and the non-canonical empty tx set test
  on ov2's Rust-overlay topologies. The tx set size test checks against
  overlay-v2's own tx set allowance. The sub-second
  getUpperBoundCloseTimeOffset test checks the offset directly, since there
  is no C++ tx queue admission. Tests that used CheckValidLedgerViewWrapper
  now use getLCLView().
- Cargo.lock: master's lockfile plus the overlay crate's packages,
  regenerated by cargo; semver-compatible duplicates take the newer
  version.
- Build: Makefile.am keeps the Rust overlay rules and adds tracy
  import-chrome; ci-build.sh keeps ASAN off and takes master's -g1.
- Needed to compile: PendingEnvelopes.cpp includes LedgerManager.h
  (master got it through the C++ OverlayManager.h); ov2's early-nomination
  and checkpoint-slot tests use ConsensusTime/ApplyTimeOffset; OverlayIPC
  tests take ledger views from the LedgerManager.
The Rust overlay pinned stellar-xdr 26.0.0, which predates protocol 30's
XDR. It strict-decodes every StellarMessage it receives, so at protocol 30
it would have:
- failed to decode millisecond close-time StellarValues
  (STELLAR_VALUE_SIGNED_MS), so tx set hashes were no longer extracted
  from SCP envelopes: no proactive tx set fetch and no tx set source
  tracking. Core still fetched on demand, but later.
- dropped transactions and tx sets containing protocol 30 XDR arms
  (CAP-0084 muxed contract addresses).

stellar-xdr 30.0.0 is generated from exactly the protocol XDR this branch
builds against (4f524bbac), and Cargo.lock already carries it for the p30
Soroban host. The crate no longer has a `curr` module, so paths change
from stellar_xdr::curr:: to stellar_xdr::.

collect_stellar_value_hash now skips empty-tx-set values (both close-time
formats). Their tx set hash is Core's synthetic empty set hash, which
Core builds locally, so fetching it from the network can never succeed.
Tests cover every StellarValue format, an ms value in a ballot, and
truncated ms values.

Message size limits stay overlay v2's own: a 16 MiB message cap in the
Rust overlay and the IPC channel, and a 10 MiB tx set allowance
(overlay/NetworkConstants.h). Master's protocol 29 cut to a 5 MiB message
cap and a 4 MiB tx set allowance (stellar#733) lives in the C++ overlay's
Peer.h/FlowControlCapacity.h. It protects that overlay from externalizing
a set upgraded peers can't download, which doesn't apply here, and it
would cap perf runs (3k SAC TPS at 2s ledgers needs about 6 MB of Soroban
bytes per ledger). The merge adapts master's tx set size test to these
constants.
Master's stellar#5423 made ARTIFICIALLY_SET_CLOSE_TIME_FOR_TESTING a millisecond
value. Two overlay v2-only tests still set it in seconds, which would now
mean 2ms and 5ms ledgers: "SCP checkpoint" (2 -> 2000) and "SCP message
capture from previous ledger" (5 -> 5000).

Master tests the upgrade to millisecond close times in UpgradesTests,
which overlay-v2-shared does not build (it needs the C++ overlay). Add
overlay v2 versions of the two tests to HerderTests:

- "millisecond close time upgrade boundary": a 4-node Rust-overlay
  network starts at protocol 29, upgrades to protocol 30 and must keep
  closing ledgers, switching to ms values exactly at the boundary. A
  fourth node is stopped before the upgrade and restarted from its
  database afterwards; it must resync across the boundary from relayed
  SCP state without catchup. The simulation runs in real time, so the
  upgrade is scheduled relative to now, and the node is removed and
  re-added since overlay v2 can't partition a single connection.
- "upgrade scheduling under sub-second ledgers": upgrade scheduling reads
  only the whole-second close time.

Also start "prepare nomination before trigger" 375ms into a second, so
the early-prepared close time carries milliseconds at protocol 30.
This affects both nominating a new transaction set, and validating the incoming transaction set. Thanks to the fact that most of the time we're either applying the ledger, or validating a transaction set, we can use the efficient CPU-pinned batch executor for this.

This speeds up the invalid transaction trimming step by ~20ms on large benchmarks, and in general should increase the transaction validation step proportionally to the number of cores.
Master's --capture-lcm/--check-lcm mode (stellar#5435, run by ci-build.sh over
[tx]) makes getTestConfig() enable ENABLE_SOROBAN_DIAGNOSTIC_EVENTS, and
Application rejects that on networked validators. Master's loopback
simulations stay RUN_STANDALONE, but overlay v2's Simulation::addNode
always clears RUN_STANDALONE (its nodes talk through Rust overlay
processes). So every simulation-based [tx] test threw at startup under
--check-lcm, e.g. "network externalizes empty-tx-set on missing value".

Simulations never capture LedgerCloseMeta (the constructor calls
disableLcmCapture()), so their nodes don't need diagnostic events.
Turn them off in addNode. Runs without LCM tracking are unaffected,
since getTestConfig() only enables the events while tracking LCM.
ci-build.sh checks [tx] against the LedgerCloseMeta golden data
(`--rng-seed 12345 --check-lcm`). With master's golden files, 32 of them
mismatch on overlay v2, while all 144 [tx] test cases pass:

- 29 Soroban files: InvokeHostFunctionTests "parallel txs|multi RO
  extensions ..." runs 0-24, "read-only bumps across threads" and two
  "restore with nonexistent key" sections, plus FrozenLedgerKeysTests
  "frozen ledger keys in Soroban footprint". These tests draw inputs from
  the global test PRNG after closing ledgers. On master, the C++
  TransactionQueue draws from the same PRNG at construction and in
  shift() on every ledger close. Overlay v2 has no C++ queue, so later
  draws differ. In "read-only bumps across threads", ledgers 1-23 are
  byte-identical and ledger 24 applies the same accounts, sequence
  numbers and fees, but with different random extendTo arguments. The
  apply path and src/rust are identical to master.
- 3 TxEnvelopeTests files for "getUpperBoundCloseTimeOffset under
  sub-second ledgers", which overlay v2 rewrote in the master merge (no
  C++ queue admission to exercise).

Regenerated from the source tree root with
`stellar-core test '[tx]' --rng-seed 12345 --capture-lcm` at the current
protocol, the only one overlay v2's CI builds. A full capture changes
exactly these 32 files: no additions, deletions or index changes.
@SirTyson
SirTyson force-pushed the merge-ms-close-time branch from cb8678e to 53a6cac Compare November 9, 2026 21:38
@socket-security

socket-security Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedcargo/​stellar-xdr@​30.0.089100100100100

View full report

@SirTyson SirTyson mentioned this pull request Nov 9, 2026
5 of 6 tasks
@SirTyson

SirTyson commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Updated (force-pushed, 2026-11-09): rebased onto the reworked #5453, which merges master and therefore already contains #5423, so the old protocol-28 backport commit is dropped. The PR is now the trigger drift-allowance commit plus four new commits (trigger recording fix, sub-second anchor tests, trigger timing metrics, a fix for a test-only delayed-emit race). See the "Update (2026-11-09)" section of the description.

Commits on top of the base:

  • 4817b1a0a Account for actual elapsed nomination time in the trigger drift allowance
  • 317e32a63 Record the nomination trigger only when nomination proceeds
  • 0de9ac1b9 Test the trigger anchor with sub-second close times
  • 5e49c5ce4 Add trigger timing metrics
  • 53a6cac68 Skip a delayed nomination emit that was already sent

@SirTyson
SirTyson force-pushed the merge-ms-close-time branch from 53a6cac to f397043 Compare November 10, 2026 21:05
The overlay-v2 experiment branches don't track master's protocol/overlay version pairing, and the check fails against overlay-v2-shared's ProtocolVersion.h.
…ance

Integrate the trigger timestamp bookkeeping from 9193e36 and the elapsed-time heuristic from 209e581. Credit steady-clock time through first ballot entry, preserve clock drift fallbacks, and cover slow rounds, delayed callbacks, missing history, and ballot overlap. Keep the existing proposal preparation policy.
triggerNextLedger started the trigger-to-ballot interval before checking
that the close time it would propose is valid. When that check fails the
node skips nomination and later enters ballot from its peers' messages, so
the interval credited that idle wait to the drift allowance.

Record the trigger after the close-time check, immediately before the
proposal is built. Add a test that skips nomination with the system clock
behind the last close time and checks that no interval is credited, and
that the same path records the trigger once the clock is back in sync.
The trigger anchor tests used whole-second close times only. With
millisecond close times the next trigger must be due exactly the target
block time after the externalized close time, and the drift allowance
boundary must hold to the millisecond, wherever in its second the close
time falls.

Add a test that tracks a millisecond close time at 0, 250 and 999 ms into
its second and checks the scheduled wait and the fallback boundary to the
millisecond. A whole-second anchor would be early by the sub-second part.
At 2 s block times it matters whether a ledger is bound by apply, by
nomination, or by main-thread delay in running the trigger. None of these
were directly observable.

Add three metrics:
- scp.timing.trigger-to-ballot (timer): local trigger to the first entry
  into ballot, i.e. proposal construction plus nomination. This is the
  interval the trigger anchor credits as its drift allowance.
- scp.trigger.lateness (timer): how long after its scheduled time the
  trigger timer callback ran.
- scp.trigger.immediate (meter): the next trigger was already due when it
  was scheduled, so the node is behind the target block time.

Metrics only; scheduling, values and ledger output are unchanged. Tests
check each metric under virtual time, including a trigger callback that
runs 300 ms late and a reschedule after the trigger is already overdue.
With the test-only ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING,
nominate() arms NOMINATION_EMIT_TIMER instead of emitting. If a peer's
envelope changes the nomination state before that timer fires,
processEnvelope emits the current votes and accepted values at once and
self-processes them. When the delayed emit then fires with nothing new, it
builds the identical statement, our own processEnvelope rejects it as not
newer, and emitNomination throws "moved to a bad state (nomination)". The
hidden "consensus close time trigger timer" simulation hit this
intermittently under parallel test load.

Route the delayed emit through emitDelayedNomination, which emits only if
the statement is newer than the last one this node processed for itself
(the same check self-processing applies). Production builds are unaffected:
the delay and the timer exist only in BUILD_TESTS builds.

Add an SCP test that replays the race deterministically: a deferred emit,
then an immediate emit triggered by a v-blocking set accepting the value,
then the deferred emit firing. It also checks that a deferred emit with
nothing sent in the meantime still broadcasts the vote.
@SirTyson
SirTyson force-pushed the merge-ms-close-time branch from f397043 to f94e02c Compare November 10, 2026 21:22
@SirTyson
SirTyson marked this pull request as ready for review November 10, 2026 22:00
@SirTyson
SirTyson merged commit 8be7506 into stellar:overlay-v2-shared Nov 10, 2026
5 checks passed
SirTyson added a commit that referenced this pull request Nov 10, 2026
## Update (2026-11-09)

Rebased onto the updated #5454 (which now sits on #5453's merge of
`master` @ 362dbc5; the overlay crate moved to `stellar-xdr` 30.0.0
there). The original commit is unchanged apart from a test import
updated for stellar-xdr 30.0.0 (`stellar_xdr::curr::` to
`stellar_xdr::`). Eight commits on top, each with tests (an independent
review found no correctness, wire-format or deadlock issues in the
original commit; its low-severity findings are fixed below):

- **Stop serving a demand after a failed write.** A response job returns
at the first write that fails after the stream retries (usually a
disconnect), instead of looking up, writing and logging a failure for
every remaining batch. The requester re-demands unanswered hashes after
its response timeout.
- **Frame responses straight from the shared transaction bytes.** Each
frame's discriminant and envelope are copied directly into the batch,
with no per-transaction allocation; a transaction larger than the batch
budget is written as two borrowed parts with no copy. Each batch is
looked up under one TX-buffer lock, released before frames are copied or
written, so local submissions on the event loop never wait for a
response copy.
- **Buffer inbound TX stream reads (64 KiB).** One QUIC read takes in
whatever has arrived, often a whole response batch, instead of two reads
per frame. The reader never waits for more data than is available.
- **Queue INVs for all peers under one lock**, for both relayed and
locally submitted transactions, instead of one batcher lock per peer.
- **Handle each received INV batch with one seen-set lock and one
tracker lock**, instead of both locks per advertised hash. Request
decisions and their order are unchanged.
- **More tests:** batching edges (an oversized transaction first, two in
a row, frames exactly at the budget, carried hits, misses); a failed
batch write retried whole on a reopened stream; buffered reads; delivery
of INV batches completed by either path; INV request decisions and
order.

Wire bytes are unchanged. Testing on the rebased branch:
- 194 overlay crate tests pass (`cargo test` in `overlay/`, 3 ignored);
`cargo fmt --check` is clean.
- The standard build succeeds, and 25 C++ overlay-IPC test cases (136
assertions) pass: `[overlay-ipc]`, `[overlay][ipc]` and the 10-node
`[overlay-ipc-large]` consensus case.
- The full suite passes on the #5458 head that contains this PR
(selftest 590 test cases, check-sorobans, check-rust with these overlay
tests, check-nondet).

The follow-up commits have not been measured separately on perfnet; the
numbers above are for the original commit on the previous base. The
perfnet run of the final #5458 head (which contains this PR) is reported
on #5458.

**Next lever (not in this PR):** a node waits up to 100 ms (checked
every 50 ms) before announcing new transactions to a peer, so each
flooding hop adds 100-150 ms before the peer can request them. A perfnet
A/B of shorter delays (50 ms and 25 ms) is the planned follow-up,
comparing tx pull latency, tx-set fetch p99 and block intervals against
bandwidth and CPU.

---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet (30 validators in
4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated
apply, 900 s load); the runs of the final #5458 and #5459 heads contain
this PR and are reported on those PRs.

# Description

This PR batches GETDATA transaction responses in the Rust overlay.
Previously, a request for many transactions created a separate task and
stream write for every response. Under load, that produced unnecessary
scheduling and stream-lock overhead, and a slow peer could leave the
entire response demand queued in memory.

We now serve each request in one asynchronous job, combining complete
transaction frames into writes of up to 64 KiB. Each transaction keeps
its existing wire framing, and the writes use the existing stream
priorities and retry policy. Transactions larger than the batch budget
are sent individually.

Here, the control image is the PR this is based on
#5454. This is the 10 tier
1, 2 second target block latency test. Median block intervals are from
the canonical node; p99 is the worst node over the full measurement
window.

## 2,000 TPS

| Metric | Control | Batched GETDATA |
|---|---:|---:|
| Median block interval | 1.993 s | 2.000 s |
| Worst-node p99 block interval | 5.090 s | 2.508 s |
| Tx-set fetch p99 | 1.720 s | 0.778 s |
| Harness verdict | FAIL | PASS |

## 3,000 TPS

| Metric | Control  | Batched GETDATA|
|---|---:|---:|
| Median block interval | 2.434 s | 2.363 s |
| Worst-node p99 block interval | 4.122 s | 4.524 s |
| Tx-set fetch p99 | 1.757 s | 1.087 s |
| Harness verdict | FAIL | FAIL |

At 2,000 TPS, the largest improvement is in the tail: worst-node p99
falls by about half, and nearly all tx-set fetches longer than two
seconds disappear. This is consistent with reducing transaction-response
write overhead and contention affecting consensus transfers.

# Checklist

- [x] Reviewed the contributing document.
- [x] One commit on top of #5454, with no merge commits. (Now nine: the
original plus eight follow-ups, see the update above.)
- [x] Formatting checked with `make format` and `cargo fmt --check`.
- [x] Release overlay build and Jenkins package build succeeded.
- [x] Relevant tests passed: 185 Rust tests (3 ignored), plus 17 C++
overlay-IPC cases / 82 assertions.
- [x] Performance comparison included above.
SirTyson added a commit that referenced this pull request Nov 10, 2026
## Update (2026-11-09)

Rebased onto the updated #5457 (the stack now includes #5453's merge of
`master` @ 362dbc5). Four commits now sit on top of the original
change.

- **Match externalized accounts in time proportional to the tx set.**
`cleanupAccounts` runs in `valueExternalized` on every node for every
ledger. It used to allocate a set entry for the source of every
transaction in the externalized set, then walk every in-use account with
a map lookup. When ledgers fall behind the offered load, the in-use
backlog grows, and that walk grew with it on the externalize path.
In-use accounts are now indexed by public key, and each externalized
transaction's source is looked up there. The cost no longer depends on
the backlog, and the lookups allocate nothing. Which accounts are
released, and when, is unchanged.
- **Stop re-registering loadgen metrics and copying envelopes per
transaction.** `execute()` used to resolve ten meters through the
metrics registry for every transaction, and copied the envelope into a
`StellarMessage` just to size it. It now uses the meters resolved at
construction and sizes the message from the envelope. The recorded
values are identical.
- **Hash externalized transactions without rebuilding their frames.** To
give the Rust overlay the full hashes of the externalized transactions,
and loadgen their source accounts, `processExternalized` built a
`TransactionFrame` for every transaction in the set on the main thread
(about 7k per ledger at 3k TPS). That copied each envelope and built one
operation frame per operation. The set's envelopes are now hashed in
place (`TxSetXDRFrame::getTransactionFullHashes`), and loadgen reads
sources straight from the envelopes. The overlay receives the same
hashes in the same order, and loadgen releases the same accounts.
- **Tests.** On this branch submission never rejects, so the recycling
test's "no rejected transactions" assertion could not fail. It now
relies on every transaction applying and on runs completing with synced
sequence numbers. It also fails fast and runs in about a third of the
time. New tests:
- "synthetic SAC load recycles source accounts in overlay-only mode"
runs perfnet's load (synthetic SAC payments, overlay-only, more
transactions than accounts). It checks completion and the
per-transaction meters, including `loadgen.txn.bytes`.
- "externalized tx set hashes and sources match transaction frames" pins
the hashes and sources from legacy and generalized sets (sequential and
parallel phases; v0, v1, muxed and fee-bump envelopes) against frames
built from the same set.

Known limit: in the `MIXED_PREGEN_*` modes, the Soroban stream fails the
run when every source account has a transaction in flight, whereas the
other modes wait. At perfnet's 33k accounts per generator, this needs a
backlog of hundreds of slow ledgers, which is longer than a
block_time_check run.

Testing:
- New and changed tests by name: the six loadgen tests (including both
new ones and the sped-up recycling test) pass.
- `[loadgen],[herder],[txset]` (partitioned `make check`): 94/94 test
cases pass.
- Full suite at the final head `e5ec7d4cb` (which contains the whole
stack): selftest 590 test cases / 2.85M assertions, check-sorobans,
check-rust, check-nondet all pass.
- An independent review found no correctness, determinism or safety
issues in the original commit or the follow-ups.

The performance tables above are for the original commit on the previous
base. Perfnet run of the final head (2026-11-09):

**Perfnet, final head `e5ec7d4cb`** (30 validators in 4 AWS regions, 10
tier-1 orgs, 3,000 SAC TPS offered from all 30 nodes, 2 s target, 150 ms
simulated apply, tx sets capped at 7,500; n = 1):
- Correct: protocol 30 with millisecond close times on all nodes; no
forks, all in sync, no invalid values/signatures, no restarts, no
overlay errors during the load.
- Verdict FAIL (window 1 p75 3.25 s on every node, p99 3.89 s),
essentially the same speed as the previous image of this PR on the same
fixture (p75 3.32 s, p99 3.92 s): about 3.1 s per ledger with full
7,500-tx ledgers and a growing backlog.
- Per ledger: ~0.3 s externalize + close (incl. 150 ms apply), **~1.05 s
tx-set build** before nomination, ~1.34 s nomination (of which ~0.33 s
tx-set fetch, ~0.34 s validation), ~0.28 s ballot. Every trigger was
already due (backlog), so the leader's early tx-set preparation never
ran and the build sits on the critical path. The new trigger metrics
(#5454) show lateness p75 0.25 ms and no prepare-start fallbacks.
- This PR's changes: loadgen submit 7-10% cheaper (mean 2.71 vs 2.91
ms); demand timeouts -21% and duplicate flood bytes -21% (both only
during the first minutes of load); neither is on the ledger critical
path in this fixture.

---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet (30 validators in
4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated
apply, 900 s load); the run of this PR's final head will be posted here.

# Description

This PR removes the linear account-selection operation from loadgen.
Previously, each transaction selected a random position in an unordered
set and walked to it. With large account pools, that traversal stalled
submission callbacks on the main thread.

Available accounts now live in a vector. Selection uses a uniform random
index, and removal swaps the selected account with the last entry before
popping it. Both operations take constant time. Accounts return to the
available pool after their ledger has applied; the test covers account
recycling and consecutive loadgen runs.

Here, the control image is the PR this is based on,
#5457. This is the 10 tier
1, 2 second target block latency test. Median block intervals are from
the canonical node; p99 is the worst node over the full measurement
window.

## 2,000 TPS

| Metric | Control | Loadgen account selection |
|---|---:|---:|
| Median block interval | 2.000 s | 1.956 s |
| Worst-node p99 block interval | 2.508 s | 2.595 s |
| Mean loadgen submission batch | 48.28 ms | 2.68 ms |
| Harness verdict | PASS | PASS |

## 3,000 TPS

| Metric | Control | Loadgen account selection |
|---|---:|---:|
| Median block interval | 2.363 s | 2.341 s |
| Worst-node p99 block interval | 4.524 s | 3.105 s |
| Mean loadgen submission batch | 88.31 ms | 3.62 ms |
| Harness verdict | FAIL | FAIL |

Submission overhead falls substantially at both loads. At 3,000 TPS,
worst-node p99 improves, but both runs still fail the harness p75
threshold. Submission-batch means use timer count/sum differences across
the ten load generators.

# Checklist

- [x] Reviewed the contributing document.
- [x] One commit on top of #5457, with no merge commits.
- [x] Formatting checked with `make format`.
- [x] Local build and Jenkins package build succeeded.
- [x] Relevant loadgen tests passed: 8 cases / 947 assertions, including
real-apply account recycling and consecutive runs.
- [x] Performance comparison included above.
SirTyson added a commit that referenced this pull request Nov 10, 2026
## Update (2026-11-10): cheaper leader build, ballot-start signature
prewarm, IPC size fix; perfnet passes on c6id

Head is now `ebdd73635`, 8 commits on top of `c26b191fa`. They fix the
two things the 2026-11-09 perfnet run found, plus a liveness bug the
next run found.

### Leader build
On perfnet the leader's 1.1 s build was on the critical path every
ledger.
- `ba4955347` **validates only what selection can use.** The leader
still pulls 2x the capacity, but validates each phase in mempool order
only until it holds 5/4 of the capacity in valid transactions. If some
are invalid it refills: twice the shortfall, then the rest. The selected
set is the same unless selection would have reached past the validated
prefix.
- `a1bffae42` hashes the candidates on the validation workers, not the
main thread. That was 18% of a cold build's instructions on hosts
without SHA extensions, such as c5d.
- `966d3bb41` precomputes the parallel builder's sort keys.
- `95be616c7` decodes mempool pulls straight from the IPC payload,
without copying.
- `2a30c1dc0` adds phase timers
`herder.txset.build-{pull,frames,trim,select,xdr,handoff,roundtrip,final-check}`
and a hidden leader-build benchmark.
- Laptop benchmark (15,000 SAC candidates, cap 7,500, 4 tasks, n=5
medians): cold build 261 -> 168 ms (-36%), warm 124 -> 84 ms (-32%).
Main-thread instructions -41% (callgrind).

### Signature prewarm from ballot start, on every validator
(`a91547bec`)
The 2026-11-09 follower prewarm covered only about half of the leader's
set. The fees are equal, so the mempool orders transactions by hash, but
selection breaks ties randomly. The leader also had no prewarm.
- Now, when slot N starts balloting, every validator verifies the
signatures of exactly the window that the N+1 build validates first.
This runs while N finishes balloting, externalizes and closes. The main
thread only makes the overlay round trip; decoding and verification run
on background tasks.
- New metrics: `herder.prewarm.{pull,duration}`,
`herder.prewarm.{txs,verified,incomplete}` and
`herder.txset.{build,validate}-sig-cache-{hit,miss}`.
- Laptop: a build after a prewarm takes 83 ms (cold 169). Received-set
validation takes 38 ms (cold 102). 0 cache misses.
- `0cf227b94`: Core's tx-set fetch timer is now `scp.fetch.txset`, with
`scp.fetch.txset-retry`. `overlay.fetch.txset` is now only the Rust
overlay's time; it had been fed by both.

### Bug fix: oversized IPC payloads (`ebdd73635`)
Perfnet found this on `a91547bec`, on c5d.
- **What happened:** 80 s into the load the whole network stopped at one
ledger. 16 of 30 nodes logged `Overlay IPC connection closed` within
0.35 s of each other. The rest then lost quorum, and all lost sync.
- **Cause:** the prewarm pulls 2 x 7,600 + 7,500 transactions (22,700
SAC payments, about 17.2 MB) once the mempool backs up. Both ends of the
Core-overlay IPC rejected payloads over 16 MiB, and Core's reader treats
a rejected length as a corrupt stream: it stops reading and never
recovers. Every validator starts the pull at the same ballot.
- **Fix:**
- The limit is now 64 MiB on both sides (`ipc::MAX_PAYLOAD_BYTES` and
`MAX_PAYLOAD_SIZE`, which must match).
- The overlay returns the longest prefix of the top transactions that
fits, with a warning, so no mempool can produce an oversized response.
- Neither side sends an over-limit payload: Rust refuses with
`PayloadTooLarge`, and Core refuses before writing anything.
  - Core now logs the type and length when it rejects one.
- This also covers a maximum-size peer tx set (`MAX_TXSET_XDR_SIZE` + 32
bytes), which could have hit the old limit too.
- **Test:** a new `[overlay-ipc]` test with the real Rust overlay. A 20
MiB top-transactions response arrives whole; with the old code the pull
returns nothing (`0 == 20`). An 80 MiB pull is truncated to the 63
highest-fee transactions, and the IPC keeps working.

### Testing
- **`a91547bec`:**
  - Laptop: new and changed tests, and the step-2 tags (149 test cases).
- Devbox: full suite `--all-versions` PASS (selftest 604 test cases /
8,638,027 assertions, check-nondet, check-rust).
- **`ebdd73635`** (devbox):
- New tests: the `[overlay-ipc]` test above, plus IPC tests for a 20 MB
message, an over-limit length (rejected, with a receive timeout so a
regression fails instead of hanging) and an over-limit send (nothing
written). Rust tests cover truncation and the send guard.
- `[ipc]` (7 test cases), `[overlay-ipc]` (20), partitioned
`[overlay]~[ipc-crossproc]` and the Rust `ipc::` tests (23): all pass.
- Two pre-existing failures reproduce on `a91547bec` without the change:
- The Rust `test_tx_not_blocked_by_scp_flood` fails its own validity
check (an SCP flood of 6.7 ms is under its 10 ms floor).
- The hidden `IPC cross-process communication` test needs an unbuilt
`overlay-stub` binary.
- Full suite `--all-versions` on `ebdd73635` (devbox): PASS. selftest
605 test cases / 8,638,079 assertions across protocol versions,
check-nondet, libsodium (77) and xdrpp (7), check-rust and the soroban
host tests (103 `test result: ok`, 0 failed). The overlay crate's own
tests run separately; see above.

### Perfnet, head `ebdd73635`
30 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered, 2
s target, 150 ms simulated apply, tx sets capped at 7,500, 900 s
measured. One run per instance type, both on the same day: c5d.2xlarge
(perfnet's default, Cascade Lake, no SHA-NI) and c6id.2xlarge (Ice Lake,
SHA-NI). Same 8 vCPU / 16 GiB.

**c6id.2xlarge: PASS.** All four windows on target, inclusion passed
(2,880,000 txs), all checks passed.

| Window | avg p75 | avg p99 | p75 range over nodes |
|---|---:|---:|---|
| 1 | 2,112 ms | 2,334 ms | 2,103-2,129 ms |
| 2 | 2,099 ms | 2,391 ms | 2,091-2,113 ms |
| 3 | 2,101 ms | 2,324 ms | 2,092-2,110 ms |
| 4 | 2,099 ms | 2,354 ms | 2,089-2,110 ms |

**c5d.2xlarge: FAIL** in window 1, but 19% faster than the previous head
and #5458. The c5d network fell behind the offered load, so sets were
full at 7,500 and the mempool backed up. That's exactly the case that
stalled `a91547bec`, and here there were no IPC disconnects or sync
losses. Correctness passed: error counters, in sync, no fork, still
closing. Inclusion was skipped because the load stopped after window 1.

| c5d, window 1 (5-min means) | #5458 head | #5459 `c26b191fa` | **#5459
`ebdd73635`** |
|---|---:|---:|---:|
| avg p75 / p99 | 3,250 / 3,891 ms | 3,264 / 3,672 ms | **2,637 / 2,920
ms** |
| Ledgers per 5 min | 96 | 96 | 119 |
| Trigger to first ballot | 2.52 s | 2.55 s | 1.94 s |
| Leader build | 1.15 s | 1.11 s | 0.68 s (candidate validation 0.20 s)
|
| Received-set validation | 0.34 s | 0.52 s | 0.33 s |
| Nomination timeouts per node | 5 | 106 | 34 |
| Ledger close | 0.229 s | 0.228 s | 0.229 s |

Both instance types, same build:

| | c5d | c6id |
|---|---:|---:|
| Ledger interval | about 2.53 s | about 2.03 s |
| Txs per ledger | 7,500 (backlog) | about 6,100 (kept up with 3,000
TPS) |
| Trigger to first ballot | 1.94 s | 1.22 s |
| Leader build | 0.68 s | 0.27 s |
| Received-set validation | 0.33 s | 0.20 s |
| Signature-cache coverage: build / validation | 89.5% / 98.7% | 87.9% /
97.8% |
| Prewarm: mean duration / main-thread pull / stopped early | 0.21 s /
22 ms / 0.1% | 0.10 s / 7 ms / 0% |
| Ledger close | 0.229 s | 0.164 s |

Notes:
- **No A/B on c6id.** #5458 has not been run on c6id, so the c6id PASS
shows this head can hold the target on current-generation x86. It
doesn't show how much of that comes from this PR. Every earlier run of
this fixture was on c5d.
- **Two regimes.** On c6id the network keeps up, so sets stay at about
6,100 transactions and triggers aren't all due at once; #5454's early
preparation runs, giving about 2 builds per ledger network-wide. On c5d
it falls behind, sets are full, and every ledger is heavier. So
per-ledger timings aren't like for like across the two.
- **What c5d still needs** (about 0.24 s to pass):
- Received-set validation still takes 0.33 s with 98.7% of its
signatures cached, so the rest of validation is now the cost.
  - About 245 ms of the c5d build is outside the phase timers.
- The remaining trigger-to-ballot time beyond build, fetch and
validation is about 0.57 s of nomination messaging across 4 regions.
- n = 1 per instance type.

---

## Previous update (2026-11-09)

_Genesis-upgrade fix, fetch robustness, follower signature prewarm._

Rebased onto the updated stack (#5453 now merges `master` @ 362dbc5:
protocol 30, millisecond close times; #5454, #5457 and #5458 updated the
same day). The PR's original four commits are kept (adapted to the new
base: stellar-xdr 30.0.0 paths, #5457's batched frame writes); ten
commits are added on top.

### Bug fix: networks starting at the genesis protocol stalled
(465f6a4)

**Root cause.** With leader-only construction (5fff543), followers no
longer build a tx set at
the trigger and must fetch the leader's. Before protocol 20 a ledger's
set is a legacy
`TransactionSet`, and the Rust overlay only carries
`GeneralizedTransactionSet`s (Core caches,
Rust decodes and Core receives only that type). So a follower could get
the leader's set only by
being promoted to leader by nomination timeouts and building it itself,
one slow round at a time.
Before this PR every validator built the identical empty legacy set (it
depends only on the
previous ledger; upgrades live in the `StellarValue`), so it was always
local. On perfnet (30 nodes)
22 nodes reached ledger 2 and stalled, 8 stayed at ledger 1. The upgrade
ledger itself
(LCL < 20) has the same problem; ledgers from protocol 20 on are
unaffected.

**Fix.** When the LCL is before protocol 20, every validator constructs
its nomination value at
the trigger again, as before; it is still voted only by round leaders.
Leader-only construction
is unchanged from protocol 20 on.

**Test.** `Rust overlay upgrades from genesis through legacy tx sets`: 4
nodes, genesis protocol 0,
four legacy ledgers at normal cadence, an upgrade straight to protocol
30, three more ledgers with
real (generalized) tx sets. It fails without the fix (a node stuck at
ledger 2) and passes with it.
Ablation on the final stack: disabling only this fix, with all the other
changes below in place,
still fails (2/2), so they don't mask the bug.

### Robustness on the same path
- 193fee3: a locally built set completes a pending fetch of the same
hash, so envelopes waiting
for it are processed at once (the IPC path had lost upstream's
ItemFetcher behaviour).
- 942a6b3: a follower promoted to leader after losing sync can still
propose; tracking is
checked at the trigger, as before. This is what made the perfnet stall
permanent.
- ab4eb14, 5964ce6: unanswered tx-set requests are retried and
reported. Core re-requests
undecided slots' sets with backoff (1, 2, 4, then every 8 s), warns on
every doubling of the
request count and counts `overlay.fetch.txset-retry`; Rust sends a retry
to a random other peer
and prunes stale reservations by slot. Core also checks a delivered
set's content hash.
- 5c159ae: test for a leader whose own build completes a pending fetch
and externalizes the
  slot inside SCP's nomination supplier.
- Rebase onto the new base: the test-only nomination override from #5481
now also hands its
substituted set to the overlay (31e6147), since this PR caches the
built set earlier.

### Performance
- 3903e4e, f2643ba: follower signature prewarm. The received-set
validation timer rose (about
158 -> 205 ms) because followers no longer validated their own mempool,
which used to warm the
signature-verification cache. Benchmark (6,000 one-signature txs, 4
tasks): 67 ms cold vs
10.5 ms warm. Followers now pull their top mempool transactions once at
the trigger and verify
master-key signatures on background tasks (at most 20 verifications per
tx; two worker threads
left free; skipped if the node already voted; stopped when validation, a
local build or
  externalization starts). They still build and compress nothing.
- c26b191: no early tx-set preparation when the trigger is already
due. Both timers fired at the
same instant in unspecified order, so a predicted leader could build its
set twice.
- 2ac0af9: no deep copies of tx-set XDR on the overlay IPC paths
(leader serializes in place,
follower decodes in place and moves into the frame). Sub-millisecond for
classic sets.

### Testing
- Laptop, on c26b191: new and changed tests (24, including the genesis
repro and the trigger,
  early-nomination and #5481 tests) and partitioned `make check` for

`[herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[nominationprotocol]~[.]`
  (143 test cases) plus the overlay's Rust tests (169 + 32 + 3) pass.
- Full suite `--all-versions` (devbox, at c26b191): selftest 598 test
cases / 8.5M assertions across protocol versions, check-sorobans,
check-rust and check-nondet pass.
- **Perfnet, head `c26b191fa`** (30 validators in 4 AWS regions, 10
tier-1 orgs, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply,
tx sets capped at 7,500; n = 1; run in parallel with the #5458 head on
the same fixture):
- **The genesis-upgrade bug is fixed on a real network:** all 30 nodes
synced in 43 s from genesis through the legacy ledgers to protocol 30
(the previous image of this PR left 22 nodes at ledger 2 and 8 at ledger
1).
- Correct under load: no forks, all in sync, no invalid
values/signatures, no restarts, no `NOT IN CACHE`/parse/cache-drop
errors; 11 tx-set fetch retries, all resolved on the first re-request.
- Leader-only construction works as designed (~1 build per ledger
network-wide instead of 30), and zstd shrinks each 7,500-tx SAC set from
~5.6 MB to ~1.0 MB (5.7x, ~7.5 ms to encode).
- Verdict FAIL at the same speed as the #5458 head (window-1 p75 3.26 s
vs 3.25 s): the leader's ~1.1 s build stays on the critical path, and
follower validation of the received set was 0.52 s vs 0.34 s on #5458
(whose followers had warmed their signature cache by building), so the
follower prewarm did not recover it at this load. Nomination round 1 (2
s timeout) times out on almost every ledger.
- Follow-up: done in the 2026-11-10 update above (leader build,
ballot-start prewarm, and the IPC size fix found by the next run).

---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet; the run of this
PR's final head will be posted in the update above.

# Description

This PR compresses transaction-set responses in the Rust overlay and
shares the encoded bytes across peer sends. Every response uses the
mandatory zstd protocol; there is no legacy/raw fallback or optional
lazy encoding state. The leader hands the final XDR to Rust before
Core's final validation, so compression can overlap that validation.
Receivers retain the original compressed bytes, decompress and verify
the set, then reuse the retained encoding when forwarding it.

A separate commit, `0ed3e540`, removes private proposals on non-leader
nodes. Followers enter nomination with a lazy value supplier and follow
received proposals without pulling the mempool or constructing a set.
First-round leaders can prepare early; a promoted leader constructs a
value only when it needs one. The driver caches that value across
retries, and externalization stops nomination even when no local value
was built. Tests cover timeout promotion, current close times, following
another leader, unavailable values, and reentrant nomination stop.

This PR builds on #5458. The
comparison below evaluates these revisions against the **original
compressed implementation of this PR**, using the same 10-tier-1,
30-validator fixture: 3,000 offered TPS, 6,000 transactions/block,
2-second target, 8 CPUs/16 GiB per validator, 60-second warmup and
900-second measurement, with simulated apply.

## 3,000 offered TPS

| Metric | Original compression | Updated leader-only path |
|---|---:|---:|
| Median block interval | 2.018 s | 2.120 s |
| Worst-node full-window p99 block interval | 3.834 s | 3.836 s |
| Tx-set fetch p99 | 0.617 s | 0.601 s |
| Completed TPS | 2,926.67 | 2,806.67 |
| Harness verdict | PASS | FAIL (ob-0 only) |

Median intervals and completed throughput use the canonical node;
worst-node p99 uses the fixed measurement window. The harness uses a
different histogram/window: `ob-0` recorded 4.989 s against its 4-second
limit, while the other 29 nodes and inclusion passed. Completed
throughput fell 4.1% and median block interval rose 5.1%; this run does
**not** establish that the revised path matches or improves the original
compression baseline. These are single cloud runs with normal host
placement.

Leader-only construction is verified: 886 builds/compressions across 421
closed ledgers, all correlated with asserted leadership, with no
follower build or received-set recompression detected. Multiple
nomination leaders can build in a ledger. Compared with the intermediate
eager implementation, local compressions fell from 13,019 to 893 in
their respective 900-second windows, roughly a 93% reduction per ledger.
Payload savings remain 80.0%. The sampled received-set validation timer
rose from about 158 ms to 205 ms versus that intermediate build; the
precise cause is not isolated by this run.

# Checklist

- [x] Reviewed the contributing document.
- [x] Kept leader-only proposal construction in its own commit on this
PR's branch.
- [x] Formatting checked with `make format` and `cargo fmt --check`.
- [x] Local build, Jenkins package/image build, and exact-head Clang/GCC
CI passed.
- [x] Rust suite: 194 passed, 3 ignored; the leader-only follow-up
leaves those validated Rust sources unchanged.
- [x] Core validation: 41 focused SCP/nomination/tx-set cases, 17
overlay IPC cases, and all 72 Herder cases covered. PostgreSQL-dependent
cases were rerun after configuring the local database fixture.
- [x] Performance comparison, runtime identity, resource checks and
sealed-artifact audit completed.
- [ ] Performance matches or improves the original compressed baseline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants