Skip to content

Batch GETDATA transaction responses - #5457

Merged
SirTyson merged 9 commits into
pr/merge-ms-close-time-basefrom
pr/batched-getdata
Nov 10, 2026
Merged

SirTyson merged 9 commits into
pr/merge-ms-close-time-basefrom
pr/batched-getdata

Conversation

@SirTyson

@SirTyson SirTyson commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Update (2026-11-09)

Rebased onto the updated #5454 (which now sits on #5453's merge of master @ 362dbc5; the overlay crate moved to stellar-xdr 30.0.0 there). The original commit is unchanged apart from a test import updated for stellar-xdr 30.0.0 (stellar_xdr::curr:: to stellar_xdr::). Eight commits on top, each with tests (an independent review found no correctness, wire-format or deadlock issues in the original commit; its low-severity findings are fixed below):

  • Stop serving a demand after a failed write. A response job returns at the first write that fails after the stream retries (usually a disconnect), instead of looking up, writing and logging a failure for every remaining batch. The requester re-demands unanswered hashes after its response timeout.
  • Frame responses straight from the shared transaction bytes. Each frame's discriminant and envelope are copied directly into the batch, with no per-transaction allocation; a transaction larger than the batch budget is written as two borrowed parts with no copy. Each batch is looked up under one TX-buffer lock, released before frames are copied or written, so local submissions on the event loop never wait for a response copy.
  • Buffer inbound TX stream reads (64 KiB). One QUIC read takes in whatever has arrived, often a whole response batch, instead of two reads per frame. The reader never waits for more data than is available.
  • Queue INVs for all peers under one lock, for both relayed and locally submitted transactions, instead of one batcher lock per peer.
  • Handle each received INV batch with one seen-set lock and one tracker lock, instead of both locks per advertised hash. Request decisions and their order are unchanged.
  • More tests: batching edges (an oversized transaction first, two in a row, frames exactly at the budget, carried hits, misses); a failed batch write retried whole on a reopened stream; buffered reads; delivery of INV batches completed by either path; INV request decisions and order.

Wire bytes are unchanged. Testing on the rebased branch:

  • 194 overlay crate tests pass (cargo test in overlay/, 3 ignored); cargo fmt --check is clean.
  • The standard build succeeds, and 25 C++ overlay-IPC test cases (136 assertions) pass: [overlay-ipc], [overlay][ipc] and the 10-node [overlay-ipc-large] consensus case.
  • The full suite passes on the Select loadgen source accounts in constant time #5458 head that contains this PR (selftest 590 test cases, check-sorobans, check-rust with these overlay tests, check-nondet).

The follow-up commits have not been measured separately on perfnet; the numbers above are for the original commit on the previous base. The perfnet run of the final #5458 head (which contains this PR) is reported on #5458.

Next lever (not in this PR): a node waits up to 100 ms (checked every 50 ms) before announcing new transactions to a peer, so each flooding hop adds 100-150 ms before the peer can request them. A perfnet A/B of shorter delays (50 ms and 25 ms) is the planned follow-up, comparing tx pull latency, tx-set fetch p99 and block intervals against bandwidth and CPU.


Original description

The numbers below come from an earlier simulation environment and are kept for history. The stack is now measured on perfnet (30 validators in 4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, 900 s load); the runs of the final #5458 and #5459 heads contain this PR and are reported on those PRs.

Description

This PR batches GETDATA transaction responses in the Rust overlay. Previously, a request for many transactions created a separate task and stream write for every response. Under load, that produced unnecessary scheduling and stream-lock overhead, and a slow peer could leave the entire response demand queued in memory.

We now serve each request in one asynchronous job, combining complete transaction frames into writes of up to 64 KiB. Each transaction keeps its existing wire framing, and the writes use the existing stream priorities and retry policy. Transactions larger than the batch budget are sent individually.

Here, the control image is the PR this is based on #5454. This is the 10 tier 1, 2 second target block latency test. Median block intervals are from the canonical node; p99 is the worst node over the full measurement window.

2,000 TPS

Metric Control Batched GETDATA
Median block interval 1.993 s 2.000 s
Worst-node p99 block interval 5.090 s 2.508 s
Tx-set fetch p99 1.720 s 0.778 s
Harness verdict FAIL PASS

3,000 TPS

Metric Control Batched GETDATA
Median block interval 2.434 s 2.363 s
Worst-node p99 block interval 4.122 s 4.524 s
Tx-set fetch p99 1.757 s 1.087 s
Harness verdict FAIL FAIL

At 2,000 TPS, the largest improvement is in the tail: worst-node p99 falls by about half, and nearly all tx-set fetches longer than two seconds disappear. This is consistent with reducing transaction-response write overhead and contention affecting consensus transfers.

Checklist

  • Reviewed the contributing document.
  • One commit on top of Merge ms close time #5454, with no merge commits. (Now nine: the original plus eight follow-ups, see the update above.)
  • Formatting checked with make format and cargo fmt --check.
  • Release overlay build and Jenkins package build succeeded.
  • Relevant tests passed: 185 Rust tests (3 ignored), plus 17 C++ overlay-IPC cases / 82 assertions.
  • Performance comparison included above.

@SirTyson
SirTyson force-pushed the pr/batched-getdata branch 2 times, most recently from cf22578 to f6ffa05 Compare September 16, 2026 03:23
@SirTyson
SirTyson force-pushed the pr/merge-ms-close-time-base branch from 49d060a to cb8678e Compare September 16, 2026 03:23
@SirTyson
SirTyson force-pushed the pr/merge-ms-close-time-base branch from cb8678e to 53a6cac Compare November 9, 2026 21:38
@SirTyson SirTyson mentioned this pull request Nov 9, 2026
5 of 6 tasks
@SirTyson

SirTyson commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Updated (force-pushed, 2026-11-09): rebased onto the updated #5454 (overlay now on stellar-xdr 30.0.0), plus eight follow-up commits from review: stop after a failed write, frame responses from the shared tx bytes without copies, buffered TX stream reads, one lock per INV batch on all three INV paths, and more tests. See the "Update (2026-11-09)" section of the description.

Commits on top of the base:

  • 16dd8888b Batch GETDATA transaction responses on the flooding stream
  • 2f68587ea Stop serving a GETDATA demand after a failed response write
  • a9973a0ef Frame GETDATA responses straight from the shared transaction bytes
  • 2708030e9 Read inbound TX streams through a 64 KiB buffer
  • 267c668b5 Queue a relayed transaction's INVs under one batcher lock
  • 6909d23eb Copy GETDATA response frames outside the TX buffer lock
  • e2f346c88 Test GETDATA batch retries and buffered TX stream reads
  • 2e8566085 Queue locally submitted INVs under one batcher lock too
  • ecfcdf789 Lock the seen set and INV tracker once per INV batch

@SirTyson
SirTyson force-pushed the pr/merge-ms-close-time-base branch from 53a6cac to f397043 Compare November 10, 2026 21:05
Port 8e926b3 onto the current timing and parallel-validation branch.
Serve each demand in one asynchronous job, combining complete transaction
frames into writes of up to 64 KiB. Preserve existing framing, priorities,
retries and standalone writes for larger transactions. Retain only the
next batch and current transaction while a peer is backpressured.

Cover multi-batch delivery, missing hashes, oversized transactions, exact
bytes and hashes, and SCP progress during a blocked TX write. Add an
ordered request barrier proving blocked responses do not retain the
entire demand; this regression fails before the change and passes after.
A batched response job kept looking up and writing the rest of a demand
after a write had already failed past send_peer_frames' retries. When a
peer disconnects, every remaining batch of every in-flight demand then
fails one by one, each with a warning, wasting buffer lookups and log
volume at the moment the node is busiest.

Move the job into serve_getdata and return at the first failed batch or
standalone write. The requester re-demands unanswered hashes from another
source after its response timeout, as it already must for a lost peer.
Abandoned hashes are logged once at debug level.

Add a test serving a multi-batch demand to a peer without streams: only
the first batch is attempted. It fails before this change.
Each served transaction was first framed into a new Vec by
to_flood_frame (allocation plus copy), then copied again into the
batch, and every hash took and released the TX buffer write lock. A
node that originates load serves each transaction to most of its
peers, so this runs at several times the network TPS.

Copy the 4-byte Transaction discriminant and the shared envelope bytes
directly into the batch (ValidatedTx::flood_frame_parts), and look up a
whole batch under one buffer lock, released before every write. A
transaction larger than the batch budget is written as two parts
straight from the shared bytes, with no copy. Wire bytes are unchanged.

Pin flood_frame_parts to the typed StellarMessage encoding, and assert
in the blocked-writer test that the buffer lock is free while a response
write waits (the test times out if the lock is held across the write).
GETDATA responses now arrive as batches of complete frames, but the
reader still pulled each frame out of QUIC with two reads (the length
prefix, then the body), and every quinn read takes the connection lock
that the connection driver and the SCP and TxSet streams also use. INV
and GETDATA frames on the same stream paid the same cost.

Wrap each inbound TX stream in a 64 KiB BufReader, so one read copies
whatever QUIC has received (often a whole response batch) and frames are
then parsed from memory. read_framed becomes generic over AsyncRead; a
frame larger than the buffer is read directly once the buffer drains.
Bytes delivered to the handlers are unchanged, and the buffer never
waits for more data than is available, so no latency is added.

Also drop the unused stream argument from handle_tx_stream_message and
handle_getdata (which no longer needs to be async).

Add a test reading a batch and oversized frames through a small
BufReader over a reader that returns short chunks and yields.
Every transaction received in a GETDATA response is announced to each
connected peer that has not advertised it. The relay took and released
the INV batcher's write lock once per peer, so on a 30-node mesh each
received transaction cost up to 28 write-lock round trips, contending
with every other peer's reader task and the housekeeping flush.

Add the entry for all peers under one lock, collect the batches that
filled up, and send them after releasing it. send_inv_batch only spawns
the write, so the batches still go out immediately. Which peers are
announced to, and the batching rules, are unchanged.

Add a test covering the relay targets (not the sender, not a peer that
already advertised the TX, every other peer) and the hand-off of a batch
that this TX completes. Export INV_BATCH_MAX_SIZE from flood for it.
Review follow-up. Since df7c3213f a response job held the TX buffer
write lock while it both looked up and copied up to 64 KiB of frames.
broadcast_tx takes the same lock inline in the event loop, so on a node
serving many peers a local submission (and any SCP command queued
behind it) could wait for several jobs' copies.

Serve each batch in two phases: under the lock, only look up the
transactions that fit the budget (a hit that does not fit is carried to
the next batch); then release it, copy the frames from the shared bytes
and write. A lone transaction above the budget is still written as two
borrowed parts. Also:
- reserve the batch exactly, so its capacity stays within the budget
  (it grew by doubling to about 117 KB before);
- count flood_fulfilled when a transaction is put into a write, so a
  failed batch no longer counts the lookup that closed it;
- share one TX_RESPONSE_BATCH_BYTES constant with the inbound read
  buffer (buffered_tx_stream);
- document lazy per-batch lookup, why a failed write ends the job, and
  that a retried batch can redeliver frames (dropped via tx_seen).

Add a test for the batching edges: an oversized TX as the first hit and
another straight after it, frames exactly at the budget, a carried
oversized TX at the end and interleaved misses, checking demand order,
exact bytes and metrics. It fails if the carried TX is dropped.
Review follow-up: two behaviours had no test.

- A batch whose write fails is rewritten whole on a reopened stream.
  Existing tests only reopened streams that were already empty. The new
  test finishes the sender's cached TX stream, serves a demand spanning
  several batches, and checks every transaction arrives once, in order,
  with no write errors and no duplicates at the receiver. It fails with
  send_peer_frames' retries disabled.
- The inbound TX stream buffer (buffered_tx_stream) takes in everything
  available in one inner read and returns each complete frame without
  waiting for more data; only a read past the available data waits.
  This pins the "adds no latency" property of 90bc974c4.
Review follow-up to 39f3d5570, which batched the INV batcher lock only
for transactions received from peers. broadcast_tx, which announces
transactions submitted by the local core, still took the write lock
once per peer, and it runs inline in the overlay event loop that also
carries SCP commands, so its lock round trips cost the most.

Move the logic into queue_inv and use it from both paths: add the entry
for every peer under one lock, then send the batches it completed after
releasing it.

The test added in 39f3d5570 only checked that a completed batch left the
batcher, so it would pass if that batch were then dropped. Add a test
over a real connection: fill the receiver's pending batch to one short
of full, complete it through the relay path and then through a local
broadcast, and wait for the receiver to demand all 1000 hashes each
time. The housekeeping flush needs 100 ms, so only the completed
batch's own send can deliver them; the test fails if that send is
removed. 30 repeated runs of the new tests: no failures.
handle_inv_batch took the tx_seen read lock and the inv_tracker write
lock once per INV entry. Every node receives about N-1 advertisements
per transaction, so on a 30-node mesh at 3k TPS that is roughly 170k
lock round trips per second, contending with every peer's reader task,
the TX relay and the housekeeping retries.

Filter the batch against tx_seen under one read lock, then record
sources and pick the first-advertised hashes under one tracker write
lock. The two locks are never held together. Decisions are the same:
seen TXs are skipped and not recorded as sources, a hash is requested
only when this is its first advertisement, repeats in a batch add the
peer once, and the demand keeps batch order. Tracker calls happen in
the same order, so its LRU state is the same. Also fix a comment that
claimed seen TXs were recorded as sources.

Add a test over a real connection: one INV batch mixing a new hash, a
seen hash, a hash first advertised by another peer, a repeat, and an
unsorted order. It checks the demand count, tracked sources and the
order in which the demanded transactions arrive. It passes before and
after this change, and fails if the seen filter or the first-advert
check is removed or the demand order is reversed. 30 repeated runs: no
failures.
@SirTyson
SirTyson force-pushed the pr/merge-ms-close-time-base branch from f397043 to f94e02c Compare November 10, 2026 21:22
@SirTyson
SirTyson marked this pull request as ready for review November 10, 2026 22:02
@SirTyson
SirTyson merged commit d782ac9 into pr/merge-ms-close-time-base Nov 10, 2026
5 checks passed
@SirTyson
SirTyson deleted the pr/batched-getdata branch November 10, 2026 22:02
SirTyson added a commit that referenced this pull request Nov 10, 2026
## Update (2026-11-09)

Rebased onto the updated #5457 (the stack now includes #5453's merge of
`master` @ 362dbc5). Four commits now sit on top of the original
change.

- **Match externalized accounts in time proportional to the tx set.**
`cleanupAccounts` runs in `valueExternalized` on every node for every
ledger. It used to allocate a set entry for the source of every
transaction in the externalized set, then walk every in-use account with
a map lookup. When ledgers fall behind the offered load, the in-use
backlog grows, and that walk grew with it on the externalize path.
In-use accounts are now indexed by public key, and each externalized
transaction's source is looked up there. The cost no longer depends on
the backlog, and the lookups allocate nothing. Which accounts are
released, and when, is unchanged.
- **Stop re-registering loadgen metrics and copying envelopes per
transaction.** `execute()` used to resolve ten meters through the
metrics registry for every transaction, and copied the envelope into a
`StellarMessage` just to size it. It now uses the meters resolved at
construction and sizes the message from the envelope. The recorded
values are identical.
- **Hash externalized transactions without rebuilding their frames.** To
give the Rust overlay the full hashes of the externalized transactions,
and loadgen their source accounts, `processExternalized` built a
`TransactionFrame` for every transaction in the set on the main thread
(about 7k per ledger at 3k TPS). That copied each envelope and built one
operation frame per operation. The set's envelopes are now hashed in
place (`TxSetXDRFrame::getTransactionFullHashes`), and loadgen reads
sources straight from the envelopes. The overlay receives the same
hashes in the same order, and loadgen releases the same accounts.
- **Tests.** On this branch submission never rejects, so the recycling
test's "no rejected transactions" assertion could not fail. It now
relies on every transaction applying and on runs completing with synced
sequence numbers. It also fails fast and runs in about a third of the
time. New tests:
- "synthetic SAC load recycles source accounts in overlay-only mode"
runs perfnet's load (synthetic SAC payments, overlay-only, more
transactions than accounts). It checks completion and the
per-transaction meters, including `loadgen.txn.bytes`.
- "externalized tx set hashes and sources match transaction frames" pins
the hashes and sources from legacy and generalized sets (sequential and
parallel phases; v0, v1, muxed and fee-bump envelopes) against frames
built from the same set.

Known limit: in the `MIXED_PREGEN_*` modes, the Soroban stream fails the
run when every source account has a transaction in flight, whereas the
other modes wait. At perfnet's 33k accounts per generator, this needs a
backlog of hundreds of slow ledgers, which is longer than a
block_time_check run.

Testing:
- New and changed tests by name: the six loadgen tests (including both
new ones and the sped-up recycling test) pass.
- `[loadgen],[herder],[txset]` (partitioned `make check`): 94/94 test
cases pass.
- Full suite at the final head `e5ec7d4cb` (which contains the whole
stack): selftest 590 test cases / 2.85M assertions, check-sorobans,
check-rust, check-nondet all pass.
- An independent review found no correctness, determinism or safety
issues in the original commit or the follow-ups.

The performance tables above are for the original commit on the previous
base. Perfnet run of the final head (2026-11-09):

**Perfnet, final head `e5ec7d4cb`** (30 validators in 4 AWS regions, 10
tier-1 orgs, 3,000 SAC TPS offered from all 30 nodes, 2 s target, 150 ms
simulated apply, tx sets capped at 7,500; n = 1):
- Correct: protocol 30 with millisecond close times on all nodes; no
forks, all in sync, no invalid values/signatures, no restarts, no
overlay errors during the load.
- Verdict FAIL (window 1 p75 3.25 s on every node, p99 3.89 s),
essentially the same speed as the previous image of this PR on the same
fixture (p75 3.32 s, p99 3.92 s): about 3.1 s per ledger with full
7,500-tx ledgers and a growing backlog.
- Per ledger: ~0.3 s externalize + close (incl. 150 ms apply), **~1.05 s
tx-set build** before nomination, ~1.34 s nomination (of which ~0.33 s
tx-set fetch, ~0.34 s validation), ~0.28 s ballot. Every trigger was
already due (backlog), so the leader's early tx-set preparation never
ran and the build sits on the critical path. The new trigger metrics
(#5454) show lateness p75 0.25 ms and no prepare-start fallbacks.
- This PR's changes: loadgen submit 7-10% cheaper (mean 2.71 vs 2.91
ms); demand timeouts -21% and duplicate flood bytes -21% (both only
during the first minutes of load); neither is on the ledger critical
path in this fixture.

---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet (30 validators in
4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated
apply, 900 s load); the run of this PR's final head will be posted here.

# Description

This PR removes the linear account-selection operation from loadgen.
Previously, each transaction selected a random position in an unordered
set and walked to it. With large account pools, that traversal stalled
submission callbacks on the main thread.

Available accounts now live in a vector. Selection uses a uniform random
index, and removal swaps the selected account with the last entry before
popping it. Both operations take constant time. Accounts return to the
available pool after their ledger has applied; the test covers account
recycling and consecutive loadgen runs.

Here, the control image is the PR this is based on,
#5457. This is the 10 tier
1, 2 second target block latency test. Median block intervals are from
the canonical node; p99 is the worst node over the full measurement
window.

## 2,000 TPS

| Metric | Control | Loadgen account selection |
|---|---:|---:|
| Median block interval | 2.000 s | 1.956 s |
| Worst-node p99 block interval | 2.508 s | 2.595 s |
| Mean loadgen submission batch | 48.28 ms | 2.68 ms |
| Harness verdict | PASS | PASS |

## 3,000 TPS

| Metric | Control | Loadgen account selection |
|---|---:|---:|
| Median block interval | 2.363 s | 2.341 s |
| Worst-node p99 block interval | 4.524 s | 3.105 s |
| Mean loadgen submission batch | 88.31 ms | 3.62 ms |
| Harness verdict | FAIL | FAIL |

Submission overhead falls substantially at both loads. At 3,000 TPS,
worst-node p99 improves, but both runs still fail the harness p75
threshold. Submission-batch means use timer count/sum differences across
the ten load generators.

# Checklist

- [x] Reviewed the contributing document.
- [x] One commit on top of #5457, with no merge commits.
- [x] Formatting checked with `make format`.
- [x] Local build and Jenkins package build succeeded.
- [x] Relevant loadgen tests passed: 8 cases / 947 assertions, including
real-apply account recycling and consecutive runs.
- [x] Performance comparison included above.
SirTyson added a commit that referenced this pull request Nov 10, 2026
## Update (2026-11-10): cheaper leader build, ballot-start signature
prewarm, IPC size fix; perfnet passes on c6id

Head is now `ebdd73635`, 8 commits on top of `c26b191fa`. They fix the
two things the 2026-11-09 perfnet run found, plus a liveness bug the
next run found.

### Leader build
On perfnet the leader's 1.1 s build was on the critical path every
ledger.
- `ba4955347` **validates only what selection can use.** The leader
still pulls 2x the capacity, but validates each phase in mempool order
only until it holds 5/4 of the capacity in valid transactions. If some
are invalid it refills: twice the shortfall, then the rest. The selected
set is the same unless selection would have reached past the validated
prefix.
- `a1bffae42` hashes the candidates on the validation workers, not the
main thread. That was 18% of a cold build's instructions on hosts
without SHA extensions, such as c5d.
- `966d3bb41` precomputes the parallel builder's sort keys.
- `95be616c7` decodes mempool pulls straight from the IPC payload,
without copying.
- `2a30c1dc0` adds phase timers
`herder.txset.build-{pull,frames,trim,select,xdr,handoff,roundtrip,final-check}`
and a hidden leader-build benchmark.
- Laptop benchmark (15,000 SAC candidates, cap 7,500, 4 tasks, n=5
medians): cold build 261 -> 168 ms (-36%), warm 124 -> 84 ms (-32%).
Main-thread instructions -41% (callgrind).

### Signature prewarm from ballot start, on every validator
(`a91547bec`)
The 2026-11-09 follower prewarm covered only about half of the leader's
set. The fees are equal, so the mempool orders transactions by hash, but
selection breaks ties randomly. The leader also had no prewarm.
- Now, when slot N starts balloting, every validator verifies the
signatures of exactly the window that the N+1 build validates first.
This runs while N finishes balloting, externalizes and closes. The main
thread only makes the overlay round trip; decoding and verification run
on background tasks.
- New metrics: `herder.prewarm.{pull,duration}`,
`herder.prewarm.{txs,verified,incomplete}` and
`herder.txset.{build,validate}-sig-cache-{hit,miss}`.
- Laptop: a build after a prewarm takes 83 ms (cold 169). Received-set
validation takes 38 ms (cold 102). 0 cache misses.
- `0cf227b94`: Core's tx-set fetch timer is now `scp.fetch.txset`, with
`scp.fetch.txset-retry`. `overlay.fetch.txset` is now only the Rust
overlay's time; it had been fed by both.

### Bug fix: oversized IPC payloads (`ebdd73635`)
Perfnet found this on `a91547bec`, on c5d.
- **What happened:** 80 s into the load the whole network stopped at one
ledger. 16 of 30 nodes logged `Overlay IPC connection closed` within
0.35 s of each other. The rest then lost quorum, and all lost sync.
- **Cause:** the prewarm pulls 2 x 7,600 + 7,500 transactions (22,700
SAC payments, about 17.2 MB) once the mempool backs up. Both ends of the
Core-overlay IPC rejected payloads over 16 MiB, and Core's reader treats
a rejected length as a corrupt stream: it stops reading and never
recovers. Every validator starts the pull at the same ballot.
- **Fix:**
- The limit is now 64 MiB on both sides (`ipc::MAX_PAYLOAD_BYTES` and
`MAX_PAYLOAD_SIZE`, which must match).
- The overlay returns the longest prefix of the top transactions that
fits, with a warning, so no mempool can produce an oversized response.
- Neither side sends an over-limit payload: Rust refuses with
`PayloadTooLarge`, and Core refuses before writing anything.
  - Core now logs the type and length when it rejects one.
- This also covers a maximum-size peer tx set (`MAX_TXSET_XDR_SIZE` + 32
bytes), which could have hit the old limit too.
- **Test:** a new `[overlay-ipc]` test with the real Rust overlay. A 20
MiB top-transactions response arrives whole; with the old code the pull
returns nothing (`0 == 20`). An 80 MiB pull is truncated to the 63
highest-fee transactions, and the IPC keeps working.

### Testing
- **`a91547bec`:**
  - Laptop: new and changed tests, and the step-2 tags (149 test cases).
- Devbox: full suite `--all-versions` PASS (selftest 604 test cases /
8,638,027 assertions, check-nondet, check-rust).
- **`ebdd73635`** (devbox):
- New tests: the `[overlay-ipc]` test above, plus IPC tests for a 20 MB
message, an over-limit length (rejected, with a receive timeout so a
regression fails instead of hanging) and an over-limit send (nothing
written). Rust tests cover truncation and the send guard.
- `[ipc]` (7 test cases), `[overlay-ipc]` (20), partitioned
`[overlay]~[ipc-crossproc]` and the Rust `ipc::` tests (23): all pass.
- Two pre-existing failures reproduce on `a91547bec` without the change:
- The Rust `test_tx_not_blocked_by_scp_flood` fails its own validity
check (an SCP flood of 6.7 ms is under its 10 ms floor).
- The hidden `IPC cross-process communication` test needs an unbuilt
`overlay-stub` binary.
- Full suite `--all-versions` on `ebdd73635` (devbox): PASS. selftest
605 test cases / 8,638,079 assertions across protocol versions,
check-nondet, libsodium (77) and xdrpp (7), check-rust and the soroban
host tests (103 `test result: ok`, 0 failed). The overlay crate's own
tests run separately; see above.

### Perfnet, head `ebdd73635`
30 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered, 2
s target, 150 ms simulated apply, tx sets capped at 7,500, 900 s
measured. One run per instance type, both on the same day: c5d.2xlarge
(perfnet's default, Cascade Lake, no SHA-NI) and c6id.2xlarge (Ice Lake,
SHA-NI). Same 8 vCPU / 16 GiB.

**c6id.2xlarge: PASS.** All four windows on target, inclusion passed
(2,880,000 txs), all checks passed.

| Window | avg p75 | avg p99 | p75 range over nodes |
|---|---:|---:|---|
| 1 | 2,112 ms | 2,334 ms | 2,103-2,129 ms |
| 2 | 2,099 ms | 2,391 ms | 2,091-2,113 ms |
| 3 | 2,101 ms | 2,324 ms | 2,092-2,110 ms |
| 4 | 2,099 ms | 2,354 ms | 2,089-2,110 ms |

**c5d.2xlarge: FAIL** in window 1, but 19% faster than the previous head
and #5458. The c5d network fell behind the offered load, so sets were
full at 7,500 and the mempool backed up. That's exactly the case that
stalled `a91547bec`, and here there were no IPC disconnects or sync
losses. Correctness passed: error counters, in sync, no fork, still
closing. Inclusion was skipped because the load stopped after window 1.

| c5d, window 1 (5-min means) | #5458 head | #5459 `c26b191fa` | **#5459
`ebdd73635`** |
|---|---:|---:|---:|
| avg p75 / p99 | 3,250 / 3,891 ms | 3,264 / 3,672 ms | **2,637 / 2,920
ms** |
| Ledgers per 5 min | 96 | 96 | 119 |
| Trigger to first ballot | 2.52 s | 2.55 s | 1.94 s |
| Leader build | 1.15 s | 1.11 s | 0.68 s (candidate validation 0.20 s)
|
| Received-set validation | 0.34 s | 0.52 s | 0.33 s |
| Nomination timeouts per node | 5 | 106 | 34 |
| Ledger close | 0.229 s | 0.228 s | 0.229 s |

Both instance types, same build:

| | c5d | c6id |
|---|---:|---:|
| Ledger interval | about 2.53 s | about 2.03 s |
| Txs per ledger | 7,500 (backlog) | about 6,100 (kept up with 3,000
TPS) |
| Trigger to first ballot | 1.94 s | 1.22 s |
| Leader build | 0.68 s | 0.27 s |
| Received-set validation | 0.33 s | 0.20 s |
| Signature-cache coverage: build / validation | 89.5% / 98.7% | 87.9% /
97.8% |
| Prewarm: mean duration / main-thread pull / stopped early | 0.21 s /
22 ms / 0.1% | 0.10 s / 7 ms / 0% |
| Ledger close | 0.229 s | 0.164 s |

Notes:
- **No A/B on c6id.** #5458 has not been run on c6id, so the c6id PASS
shows this head can hold the target on current-generation x86. It
doesn't show how much of that comes from this PR. Every earlier run of
this fixture was on c5d.
- **Two regimes.** On c6id the network keeps up, so sets stay at about
6,100 transactions and triggers aren't all due at once; #5454's early
preparation runs, giving about 2 builds per ledger network-wide. On c5d
it falls behind, sets are full, and every ledger is heavier. So
per-ledger timings aren't like for like across the two.
- **What c5d still needs** (about 0.24 s to pass):
- Received-set validation still takes 0.33 s with 98.7% of its
signatures cached, so the rest of validation is now the cost.
  - About 245 ms of the c5d build is outside the phase timers.
- The remaining trigger-to-ballot time beyond build, fetch and
validation is about 0.57 s of nomination messaging across 4 regions.
- n = 1 per instance type.

---

## Previous update (2026-11-09)

_Genesis-upgrade fix, fetch robustness, follower signature prewarm._

Rebased onto the updated stack (#5453 now merges `master` @ 362dbc5:
protocol 30, millisecond close times; #5454, #5457 and #5458 updated the
same day). The PR's original four commits are kept (adapted to the new
base: stellar-xdr 30.0.0 paths, #5457's batched frame writes); ten
commits are added on top.

### Bug fix: networks starting at the genesis protocol stalled
(465f6a4)

**Root cause.** With leader-only construction (5fff543), followers no
longer build a tx set at
the trigger and must fetch the leader's. Before protocol 20 a ledger's
set is a legacy
`TransactionSet`, and the Rust overlay only carries
`GeneralizedTransactionSet`s (Core caches,
Rust decodes and Core receives only that type). So a follower could get
the leader's set only by
being promoted to leader by nomination timeouts and building it itself,
one slow round at a time.
Before this PR every validator built the identical empty legacy set (it
depends only on the
previous ledger; upgrades live in the `StellarValue`), so it was always
local. On perfnet (30 nodes)
22 nodes reached ledger 2 and stalled, 8 stayed at ledger 1. The upgrade
ledger itself
(LCL < 20) has the same problem; ledgers from protocol 20 on are
unaffected.

**Fix.** When the LCL is before protocol 20, every validator constructs
its nomination value at
the trigger again, as before; it is still voted only by round leaders.
Leader-only construction
is unchanged from protocol 20 on.

**Test.** `Rust overlay upgrades from genesis through legacy tx sets`: 4
nodes, genesis protocol 0,
four legacy ledgers at normal cadence, an upgrade straight to protocol
30, three more ledgers with
real (generalized) tx sets. It fails without the fix (a node stuck at
ledger 2) and passes with it.
Ablation on the final stack: disabling only this fix, with all the other
changes below in place,
still fails (2/2), so they don't mask the bug.

### Robustness on the same path
- 193fee3: a locally built set completes a pending fetch of the same
hash, so envelopes waiting
for it are processed at once (the IPC path had lost upstream's
ItemFetcher behaviour).
- 942a6b3: a follower promoted to leader after losing sync can still
propose; tracking is
checked at the trigger, as before. This is what made the perfnet stall
permanent.
- ab4eb14, 5964ce6: unanswered tx-set requests are retried and
reported. Core re-requests
undecided slots' sets with backoff (1, 2, 4, then every 8 s), warns on
every doubling of the
request count and counts `overlay.fetch.txset-retry`; Rust sends a retry
to a random other peer
and prunes stale reservations by slot. Core also checks a delivered
set's content hash.
- 5c159ae: test for a leader whose own build completes a pending fetch
and externalizes the
  slot inside SCP's nomination supplier.
- Rebase onto the new base: the test-only nomination override from #5481
now also hands its
substituted set to the overlay (31e6147), since this PR caches the
built set earlier.

### Performance
- 3903e4e, f2643ba: follower signature prewarm. The received-set
validation timer rose (about
158 -> 205 ms) because followers no longer validated their own mempool,
which used to warm the
signature-verification cache. Benchmark (6,000 one-signature txs, 4
tasks): 67 ms cold vs
10.5 ms warm. Followers now pull their top mempool transactions once at
the trigger and verify
master-key signatures on background tasks (at most 20 verifications per
tx; two worker threads
left free; skipped if the node already voted; stopped when validation, a
local build or
  externalization starts). They still build and compress nothing.
- c26b191: no early tx-set preparation when the trigger is already
due. Both timers fired at the
same instant in unspecified order, so a predicted leader could build its
set twice.
- 2ac0af9: no deep copies of tx-set XDR on the overlay IPC paths
(leader serializes in place,
follower decodes in place and moves into the frame). Sub-millisecond for
classic sets.

### Testing
- Laptop, on c26b191: new and changed tests (24, including the genesis
repro and the trigger,
  early-nomination and #5481 tests) and partitioned `make check` for

`[herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[nominationprotocol]~[.]`
  (143 test cases) plus the overlay's Rust tests (169 + 32 + 3) pass.
- Full suite `--all-versions` (devbox, at c26b191): selftest 598 test
cases / 8.5M assertions across protocol versions, check-sorobans,
check-rust and check-nondet pass.
- **Perfnet, head `c26b191fa`** (30 validators in 4 AWS regions, 10
tier-1 orgs, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply,
tx sets capped at 7,500; n = 1; run in parallel with the #5458 head on
the same fixture):
- **The genesis-upgrade bug is fixed on a real network:** all 30 nodes
synced in 43 s from genesis through the legacy ledgers to protocol 30
(the previous image of this PR left 22 nodes at ledger 2 and 8 at ledger
1).
- Correct under load: no forks, all in sync, no invalid
values/signatures, no restarts, no `NOT IN CACHE`/parse/cache-drop
errors; 11 tx-set fetch retries, all resolved on the first re-request.
- Leader-only construction works as designed (~1 build per ledger
network-wide instead of 30), and zstd shrinks each 7,500-tx SAC set from
~5.6 MB to ~1.0 MB (5.7x, ~7.5 ms to encode).
- Verdict FAIL at the same speed as the #5458 head (window-1 p75 3.26 s
vs 3.25 s): the leader's ~1.1 s build stays on the critical path, and
follower validation of the received set was 0.52 s vs 0.34 s on #5458
(whose followers had warmed their signature cache by building), so the
follower prewarm did not recover it at this load. Nomination round 1 (2
s timeout) times out on almost every ledger.
- Follow-up: done in the 2026-11-10 update above (leader build,
ballot-start prewarm, and the IPC size fix found by the next run).

---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet; the run of this
PR's final head will be posted in the update above.

# Description

This PR compresses transaction-set responses in the Rust overlay and
shares the encoded bytes across peer sends. Every response uses the
mandatory zstd protocol; there is no legacy/raw fallback or optional
lazy encoding state. The leader hands the final XDR to Rust before
Core's final validation, so compression can overlap that validation.
Receivers retain the original compressed bytes, decompress and verify
the set, then reuse the retained encoding when forwarding it.

A separate commit, `0ed3e540`, removes private proposals on non-leader
nodes. Followers enter nomination with a lazy value supplier and follow
received proposals without pulling the mempool or constructing a set.
First-round leaders can prepare early; a promoted leader constructs a
value only when it needs one. The driver caches that value across
retries, and externalization stops nomination even when no local value
was built. Tests cover timeout promotion, current close times, following
another leader, unavailable values, and reentrant nomination stop.

This PR builds on #5458. The
comparison below evaluates these revisions against the **original
compressed implementation of this PR**, using the same 10-tier-1,
30-validator fixture: 3,000 offered TPS, 6,000 transactions/block,
2-second target, 8 CPUs/16 GiB per validator, 60-second warmup and
900-second measurement, with simulated apply.

## 3,000 offered TPS

| Metric | Original compression | Updated leader-only path |
|---|---:|---:|
| Median block interval | 2.018 s | 2.120 s |
| Worst-node full-window p99 block interval | 3.834 s | 3.836 s |
| Tx-set fetch p99 | 0.617 s | 0.601 s |
| Completed TPS | 2,926.67 | 2,806.67 |
| Harness verdict | PASS | FAIL (ob-0 only) |

Median intervals and completed throughput use the canonical node;
worst-node p99 uses the fixed measurement window. The harness uses a
different histogram/window: `ob-0` recorded 4.989 s against its 4-second
limit, while the other 29 nodes and inclusion passed. Completed
throughput fell 4.1% and median block interval rose 5.1%; this run does
**not** establish that the revised path matches or improves the original
compression baseline. These are single cloud runs with normal host
placement.

Leader-only construction is verified: 886 builds/compressions across 421
closed ledgers, all correlated with asserted leadership, with no
follower build or received-set recompression detected. Multiple
nomination leaders can build in a ledger. Compared with the intermediate
eager implementation, local compressions fell from 13,019 to 893 in
their respective 900-second windows, roughly a 93% reduction per ledger.
Payload savings remain 80.0%. The sampled received-set validation timer
rose from about 158 ms to 205 ms versus that intermediate build; the
precise cause is not isolated by this run.

# Checklist

- [x] Reviewed the contributing document.
- [x] Kept leader-only proposal construction in its own commit on this
PR's branch.
- [x] Formatting checked with `make format` and `cargo fmt --check`.
- [x] Local build, Jenkins package/image build, and exact-head Clang/GCC
CI passed.
- [x] Rust suite: 194 passed, 3 ignored; the leader-only follow-up
leaves those validated Rust sources unchanged.
- [x] Core validation: 41 focused SCP/nomination/tx-set cases, 17
overlay IPC cases, and all 72 Herder cases covered. PostgreSQL-dependent
cases were rerun after configuring the local database fixture.
- [x] Performance comparison, runtime identity, resource checks and
sealed-artifact audit completed.
- [ ] Performance matches or improves the original compressed baseline.
SirTyson added a commit that referenced this pull request Nov 10, 2026
…der schedule (#5496)

Brings the stacked PRs that were merged into
`pr/merge-ms-close-time-base` into `overlay-v2-shared`:

- #5457 Batch GETDATA transaction responses
- #5458 Select loadgen source accounts in constant time
- #5459 Compress and share transaction set responses
- #5495 Prototype: replace SCP nomination with a deterministic leader
schedule
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant