Skip to content

Tx set validity hardening - #5481

Merged
SirTyson merged 1 commit into
stellar:masterfrom
SirTyson:tx-set-validity-hardening
Oct 9, 2026
Merged

SirTyson merged 1 commit into
stellar:masterfrom
SirTyson:tx-set-validity-hardening

Conversation

@SirTyson

@SirTyson SirTyson commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Description

Resolves https://github.com/stellar/stellar-core-internal/issues/737

I also make it such that we do an additional check in catchup, where we make sure downloaded tx sets don't contain both the new and legacy tx set format at the same time.

Rebased on #5478.

Checklist

  • Reviewed the contributing document
  • Rebased on top of master (no merge commits)
  • Ran clang-format v8.0.0 (via make format or the Visual Studio extension)
  • Compiles
  • Ran all tests
  • If change impacts performance, include supporting evidence per the performance document

@SirTyson
SirTyson requested a review from graydon October 7, 2026 17:10
graydon
graydon previously approved these changes Oct 7, 2026
@SirTyson
SirTyson force-pushed the tx-set-validity-hardening branch from d198bbe to de650b8 Compare October 8, 2026 23:37
@SirTyson
SirTyson marked this pull request as ready for review October 8, 2026 23:37
Copilot AI balanced review requested due to automatic review settings October 8, 2026 23:37
@SirTyson
SirTyson force-pushed the tx-set-validity-hardening branch from de650b8 to 44445ea Compare October 8, 2026 23:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Empty sequential Soroban sets remain noncanonical but valid, and the stated base PR’s XDR identity check is regressed.

1 open finding
What changed in this PR

Hardens transaction-set validation and catchup while incorporating the protocol 30 promotion.

Changes:

  • Rejects noncanonical empty Soroban phases and malformed mixed-format history entries.
  • Adds consensus and catchup regression coverage.
  • Promotes protocol/Soroban versions and regenerates current/next golden data.
File Description
Cargo.lock Updates quorum-analyzer dependencies.
src/​Makefile.am Promotes the p30 Soroban host.
src/​catchup/​ApplyCheckpointWork.cpp Validates replayed transaction-set formats and canonical empty sets.
src/​herder/​HerderImpl.cpp Adds test-only nomination overrides.
src/​herder/​HerderImpl.h Declares the nomination override hook.
src/​herder/​TxSetFrame.cpp Adds protocol-gated empty-phase validation.
src/​herder/​test/​HerderTests.cpp Tests network rejection of noncanonical empty sets.
src/​herder/​test/​TxSetTests.cpp Extends structural and canonicalization tests.
src/​history/​test/​HistoryTests.cpp Tests rejection of forged history entries.
src/​main/​Config.cpp Promotes the current protocol to 30.
src/​main/​main.cpp Disables the XDR identity check.
src/​rust/​Cargo.toml Updates p30 and quorum-analyzer dependencies.
src/​rust/​src/​soroban_module_cache.rs Adds p30/p31 cache routing.
src/​rust/​src/​soroban_proto_all.rs Promotes p30 as the current host.
src/​transactions/​test/​InvokeHostFunctionTests.cpp Adjusts module-cache expectations.
src/​util/​ProtocolVersion.h Adds protocol 31 and the canonical-empty-set gate.
test-tx-meta-baseline-current/​BeginSponsoringFutureReservesTests.json Records protocol 30 baselines.
test-tx-meta-baseline-current/​BumpSequenceTests.json Records protocol 30 baselines.
test-tx-meta-baseline-current/​ClawbackClaimableBalanceTests.json Records protocol 30 baselines.
test-tx-meta-baseline-current/​ClawbackTests.json Records protocol 30 baselines.
test-tx-meta-baseline-current/​CreateAccountTests.json Records protocol 30 baselines.
test-tx-meta-baseline-current/​EndSponsoringFutureReservesTests.json Records protocol 30 baselines.
test-tx-meta-baseline-current/​EventTests.json Records protocol 30 baselines.
test-tx-meta-baseline-current/​FeeBumpTransactionTests.json Records protocol 30 baselines.
test-tx-meta-baseline-current/​FrozenLedgerKeysTests.json Advances current baseline configuration.
test-tx-meta-baseline-current/​InflationTests.json Advances current baseline configuration.
test-tx-meta-baseline-current/​LiquidityPoolDepositTests.json Records protocol 30 baselines.
test-tx-meta-baseline-current/​ManageDataTests.json Records protocol 30 baselines.
test-tx-meta-baseline-current/​RevokeSponsorshipTests.json Records protocol 30 baselines.
test-tx-meta-baseline-current/​SetOptionsTests.json Records protocol 30 baselines.
test-tx-meta-baseline-current/​TxResultsTests.json Records protocol 30 baselines.
test-tx-meta-baseline-next/​BeginSponsoringFutureReservesTests.json Records protocol 31 baselines.
test-tx-meta-baseline-next/​BumpSequenceTests.json Records protocol 31 baselines.
test-tx-meta-baseline-next/​ClawbackClaimableBalanceTests.json Records protocol 31 baselines.
test-tx-meta-baseline-next/​ClawbackTests.json Records protocol 31 baselines.
test-tx-meta-baseline-next/​CreateAccountTests.json Records protocol 31 baselines.
test-tx-meta-baseline-next/​EndSponsoringFutureReservesTests.json Records protocol 31 baselines.
test-tx-meta-baseline-next/​EventTests.json Records protocol 31 baselines.
test-tx-meta-baseline-next/​FeeBumpTransactionTests.json Records protocol 31 baselines.
test-tx-meta-baseline-next/​FrozenLedgerKeysTests.json Advances next baseline configuration.
test-tx-meta-baseline-next/​InflationTests.json Advances next baseline configuration.
test-tx-meta-baseline-next/​LiquidityPoolDepositTests.json Records protocol 31 baselines.
test-tx-meta-baseline-next/​ManageDataTests.json Records protocol 31 baselines.
test-tx-meta-baseline-next/​RevokeSponsorshipTests.json Records protocol 31 baselines.
test-tx-meta-baseline-next/​SetOptionsTests.json Records protocol 31 baselines.
test-tx-meta-baseline-next/​TxResultsTests.json Records protocol 31 baselines.
test-lcm-current/​BeginSponsoringFutureReservesTests/​index.json Reindexes protocol 30 LCM goldens.
test-lcm-current/​BeginSponsoringFutureReservesTests/​16ca002da0d1fd67.xdr Adds a protocol 30 LCM golden.
test-lcm-current/​BeginSponsoringFutureReservesTests/​1c7fb8566ea79e53.xdr Adds a protocol 30 LCM golden.
test-lcm-current/​BeginSponsoringFutureReservesTests/​24cec2c5f1acf638.xdr Adds a protocol 30 LCM golden.
test-lcm-current/​BeginSponsoringFutureReservesTests/​5a03a46b7b48bb06.xdr Adds a protocol 30 LCM golden.
test-lcm-current/​BeginSponsoringFutureReservesTests/​6659d8dd4f89982c.xdr Adds a protocol 30 LCM golden.
test-lcm-current/​FrozenLedgerKeysTests/​index.json Advances the current LCM protocol.
test-lcm-current/​MergeTests/​index.json Reindexes protocol 30 merge metadata.
test-lcm-current/​MergeTests/​6066e53439f6f65d.xdr Adds protocol 30 merge metadata.
test-lcm-current/​OfferTests/​index.json Reindexes protocol 30 offer metadata.
test-lcm-current/​PathPaymentTests/​index.json Reindexes protocol 30 path-payment metadata.
test-lcm-current/​TxEnvelopeTests/​index.json Reindexes protocol 30 envelope metadata.
test-lcm-current/​TxResultsTests/​index.json Reindexes protocol 30 result metadata.
test-lcm-next/​BeginSponsoringFutureReservesTests/​index.json Reindexes protocol 31 LCM goldens.
test-lcm-next/​FrozenLedgerKeysTests/​index.json Advances the next LCM protocol.
test-lcm-next/​MergeTests/​index.json Reindexes protocol 31 merge metadata.
test-lcm-next/​OfferTests/​index.json Reindexes protocol 31 offer metadata.
test-lcm-next/​PathPaymentTests/​index.json Reindexes protocol 31 path-payment metadata.
test-lcm-next/​TxEnvelopeTests/​index.json Reindexes protocol 31 envelope metadata.
test-lcm-next/​TxResultsTests/​index.json Reindexes protocol 31 result metadata.

🧠 Review effort: Balanced


💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/herder/TxSetFrame.cpp Resolved
@SirTyson
SirTyson requested review from graydon and a balanced review from Copilot October 8, 2026 23:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Consensus-validity and archive-replay compatibility changes warrant final human review despite no additional concrete issues found.

1 open finding

🧠 Review effort: Balanced

graydon
graydon previously approved these changes Oct 8, 2026
Copilot AI balanced review requested due to automatic review settings October 9, 2026 20:42
@SirTyson
SirTyson force-pushed the tx-set-validity-hardening branch from 44445ea to c285b15 Compare October 9, 2026 20:42
@SirTyson
SirTyson requested a review from sisuresh October 9, 2026 20:43
@SirTyson
SirTyson enabled auto-merge October 9, 2026 20:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The mixed-format catchup check can be bypassed by an empty legacy set with a populated previous-ledger hash.

2 open findings

🧠 Review effort: Balanced

Comment thread src/catchup/ApplyCheckpointWork.cpp Resolved
@SirTyson
SirTyson added this pull request to the merge queue Oct 9, 2026
Merged via the queue into stellar:master with commit 0e49751 Oct 9, 2026
45 of 64 checks passed
@SirTyson
SirTyson deleted the tx-set-validity-hardening branch October 9, 2026 23:18
SirTyson added a commit to SirTyson/stellar-core that referenced this pull request Nov 9, 2026
Brings overlay-v2-shared up to date with master. The previous master
merge (b90d48e, stellar#5428) was recorded with master's tip 100cc38 missing
from its parents, so this merge used 100cc38 as its base (temporary
graft of b90d48e, removed afterwards).

Notable master changes brought in:
- Protocol 30: CURRENT_LEDGER_PROTOCOL_VERSION follows
  MAX_SUPPORTED_PROTOCOL_VERSION; CAP-0084 and CAP-0087 ungated, p29 and
  p30 Soroban hosts (stellar#5472, stellar#5478, stellar#5484, stellar#5483).
- Millisecond-resolution close times at protocol 30 (stellar#5423):
  ConsensusTime/ApplyTime types, STELLAR_VALUE_SIGNED_MS and
  STELLAR_VALUE_EMPTY_TX_SET_MS.
- Tx set validity hardening (stellar#5481), lower max message size in protocol
  29 (stellar#733), sub-second ledger edge-case fixes.
- Simplified ledger view hierarchy (stellar#5461); the overlay-only-mode
  sequence number skip now lives in TransactionFrame::checkValid.
- Banned accounts deprecated (CAP-0077 is active).
- Apply-path performance work (stellar#5402, stellar#5411, stellar#5414, stellar#5415, stellar#5418, stellar#5421,
  stellar#5436, stellar#5286), apply-load tx set construction mode (stellar#5448), --check-lcm
  (stellar#5435), v29.0.0 release merge.

Conflict resolution:
- Files overlay-v2-shared deleted and master modified (the C++ overlay,
  TransactionQueue, TransactionQueueTests, UpgradesTests, private/Horizon/RPC
  workflows): kept deleted. Master's changes to them are C++ overlay
  peer/flow-control logic, tx queue admission and tests for those.
- Herder: kept the Rust-overlay submission path (recvTransaction returns
  TxSubmitStatus) and dropped the deprecated `force` parameter as master did.
  triggerNextLedger keeps the prepared/built tx set path from the Rust
  mempool, now with ConsensusTime close times and master's apply-time
  close-time offset. The early tx set preparation path (prepareTxSet,
  buildTxSet, setupTriggerNextLedger) takes ConsensusTime. Kept master's
  mNominatedTxSetOverride test hook (applied before the tx set is cached in
  the Rust overlay) and the herder.txset.build timer (now timing
  buildTxSet). Dropped the banned-accounts setup and tx queue maintenance.
- AppConnector: dropped both ov2's maybeUpdateImmutableLedgerView and
  master's C++-overlay getOverlayThreadSnapshot.
- LoadGenerator: master's CachedTestAccountPtr with ov2's TxSubmitStatus.
- Tests: kept ov2's removal of tests that need the C++ overlay, standalone
  mode or the tx queue (standalone, manualclose, flooding, tx queue limits,
  far-future slot cleanup, herderExternalizesValuesWithProtocol, MAX_SAC_TPS,
  custom token, soroswap and noisy binary search apply-load tests). Added
  master's ms close time HerderTests and the non-canonical empty tx set test
  on ov2's Rust-overlay topologies. The tx set size test checks against
  overlay-v2's own tx set allowance. The sub-second
  getUpperBoundCloseTimeOffset test checks the offset directly, since there
  is no C++ tx queue admission. Tests that used CheckValidLedgerViewWrapper
  now use getLCLView().
- Cargo.lock: master's lockfile plus the overlay crate's packages,
  regenerated by cargo; semver-compatible duplicates take the newer
  version.
- Build: Makefile.am keeps the Rust overlay rules and adds tracy
  import-chrome; ci-build.sh keeps ASAN off and takes master's -g1.
- Needed to compile: PendingEnvelopes.cpp includes LedgerManager.h
  (master got it through the C++ OverlayManager.h); ov2's early-nomination
  and checkpoint-slot tests use ConsensusTime/ApplyTimeOffset; OverlayIPC
  tests take ledger views from the LedgerManager.
SirTyson added a commit that referenced this pull request Nov 10, 2026
## Update (2026-11-10): cheaper leader build, ballot-start signature
prewarm, IPC size fix; perfnet passes on c6id

Head is now `ebdd73635`, 8 commits on top of `c26b191fa`. They fix the
two things the 2026-11-09 perfnet run found, plus a liveness bug the
next run found.

### Leader build
On perfnet the leader's 1.1 s build was on the critical path every
ledger.
- `ba4955347` **validates only what selection can use.** The leader
still pulls 2x the capacity, but validates each phase in mempool order
only until it holds 5/4 of the capacity in valid transactions. If some
are invalid it refills: twice the shortfall, then the rest. The selected
set is the same unless selection would have reached past the validated
prefix.
- `a1bffae42` hashes the candidates on the validation workers, not the
main thread. That was 18% of a cold build's instructions on hosts
without SHA extensions, such as c5d.
- `966d3bb41` precomputes the parallel builder's sort keys.
- `95be616c7` decodes mempool pulls straight from the IPC payload,
without copying.
- `2a30c1dc0` adds phase timers
`herder.txset.build-{pull,frames,trim,select,xdr,handoff,roundtrip,final-check}`
and a hidden leader-build benchmark.
- Laptop benchmark (15,000 SAC candidates, cap 7,500, 4 tasks, n=5
medians): cold build 261 -> 168 ms (-36%), warm 124 -> 84 ms (-32%).
Main-thread instructions -41% (callgrind).

### Signature prewarm from ballot start, on every validator
(`a91547bec`)
The 2026-11-09 follower prewarm covered only about half of the leader's
set. The fees are equal, so the mempool orders transactions by hash, but
selection breaks ties randomly. The leader also had no prewarm.
- Now, when slot N starts balloting, every validator verifies the
signatures of exactly the window that the N+1 build validates first.
This runs while N finishes balloting, externalizes and closes. The main
thread only makes the overlay round trip; decoding and verification run
on background tasks.
- New metrics: `herder.prewarm.{pull,duration}`,
`herder.prewarm.{txs,verified,incomplete}` and
`herder.txset.{build,validate}-sig-cache-{hit,miss}`.
- Laptop: a build after a prewarm takes 83 ms (cold 169). Received-set
validation takes 38 ms (cold 102). 0 cache misses.
- `0cf227b94`: Core's tx-set fetch timer is now `scp.fetch.txset`, with
`scp.fetch.txset-retry`. `overlay.fetch.txset` is now only the Rust
overlay's time; it had been fed by both.

### Bug fix: oversized IPC payloads (`ebdd73635`)
Perfnet found this on `a91547bec`, on c5d.
- **What happened:** 80 s into the load the whole network stopped at one
ledger. 16 of 30 nodes logged `Overlay IPC connection closed` within
0.35 s of each other. The rest then lost quorum, and all lost sync.
- **Cause:** the prewarm pulls 2 x 7,600 + 7,500 transactions (22,700
SAC payments, about 17.2 MB) once the mempool backs up. Both ends of the
Core-overlay IPC rejected payloads over 16 MiB, and Core's reader treats
a rejected length as a corrupt stream: it stops reading and never
recovers. Every validator starts the pull at the same ballot.
- **Fix:**
- The limit is now 64 MiB on both sides (`ipc::MAX_PAYLOAD_BYTES` and
`MAX_PAYLOAD_SIZE`, which must match).
- The overlay returns the longest prefix of the top transactions that
fits, with a warning, so no mempool can produce an oversized response.
- Neither side sends an over-limit payload: Rust refuses with
`PayloadTooLarge`, and Core refuses before writing anything.
  - Core now logs the type and length when it rejects one.
- This also covers a maximum-size peer tx set (`MAX_TXSET_XDR_SIZE` + 32
bytes), which could have hit the old limit too.
- **Test:** a new `[overlay-ipc]` test with the real Rust overlay. A 20
MiB top-transactions response arrives whole; with the old code the pull
returns nothing (`0 == 20`). An 80 MiB pull is truncated to the 63
highest-fee transactions, and the IPC keeps working.

### Testing
- **`a91547bec`:**
  - Laptop: new and changed tests, and the step-2 tags (149 test cases).
- Devbox: full suite `--all-versions` PASS (selftest 604 test cases /
8,638,027 assertions, check-nondet, check-rust).
- **`ebdd73635`** (devbox):
- New tests: the `[overlay-ipc]` test above, plus IPC tests for a 20 MB
message, an over-limit length (rejected, with a receive timeout so a
regression fails instead of hanging) and an over-limit send (nothing
written). Rust tests cover truncation and the send guard.
- `[ipc]` (7 test cases), `[overlay-ipc]` (20), partitioned
`[overlay]~[ipc-crossproc]` and the Rust `ipc::` tests (23): all pass.
- Two pre-existing failures reproduce on `a91547bec` without the change:
- The Rust `test_tx_not_blocked_by_scp_flood` fails its own validity
check (an SCP flood of 6.7 ms is under its 10 ms floor).
- The hidden `IPC cross-process communication` test needs an unbuilt
`overlay-stub` binary.
- Full suite `--all-versions` on `ebdd73635` (devbox): PASS. selftest
605 test cases / 8,638,079 assertions across protocol versions,
check-nondet, libsodium (77) and xdrpp (7), check-rust and the soroban
host tests (103 `test result: ok`, 0 failed). The overlay crate's own
tests run separately; see above.

### Perfnet, head `ebdd73635`
30 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered, 2
s target, 150 ms simulated apply, tx sets capped at 7,500, 900 s
measured. One run per instance type, both on the same day: c5d.2xlarge
(perfnet's default, Cascade Lake, no SHA-NI) and c6id.2xlarge (Ice Lake,
SHA-NI). Same 8 vCPU / 16 GiB.

**c6id.2xlarge: PASS.** All four windows on target, inclusion passed
(2,880,000 txs), all checks passed.

| Window | avg p75 | avg p99 | p75 range over nodes |
|---|---:|---:|---|
| 1 | 2,112 ms | 2,334 ms | 2,103-2,129 ms |
| 2 | 2,099 ms | 2,391 ms | 2,091-2,113 ms |
| 3 | 2,101 ms | 2,324 ms | 2,092-2,110 ms |
| 4 | 2,099 ms | 2,354 ms | 2,089-2,110 ms |

**c5d.2xlarge: FAIL** in window 1, but 19% faster than the previous head
and #5458. The c5d network fell behind the offered load, so sets were
full at 7,500 and the mempool backed up. That's exactly the case that
stalled `a91547bec`, and here there were no IPC disconnects or sync
losses. Correctness passed: error counters, in sync, no fork, still
closing. Inclusion was skipped because the load stopped after window 1.

| c5d, window 1 (5-min means) | #5458 head | #5459 `c26b191fa` | **#5459
`ebdd73635`** |
|---|---:|---:|---:|
| avg p75 / p99 | 3,250 / 3,891 ms | 3,264 / 3,672 ms | **2,637 / 2,920
ms** |
| Ledgers per 5 min | 96 | 96 | 119 |
| Trigger to first ballot | 2.52 s | 2.55 s | 1.94 s |
| Leader build | 1.15 s | 1.11 s | 0.68 s (candidate validation 0.20 s)
|
| Received-set validation | 0.34 s | 0.52 s | 0.33 s |
| Nomination timeouts per node | 5 | 106 | 34 |
| Ledger close | 0.229 s | 0.228 s | 0.229 s |

Both instance types, same build:

| | c5d | c6id |
|---|---:|---:|
| Ledger interval | about 2.53 s | about 2.03 s |
| Txs per ledger | 7,500 (backlog) | about 6,100 (kept up with 3,000
TPS) |
| Trigger to first ballot | 1.94 s | 1.22 s |
| Leader build | 0.68 s | 0.27 s |
| Received-set validation | 0.33 s | 0.20 s |
| Signature-cache coverage: build / validation | 89.5% / 98.7% | 87.9% /
97.8% |
| Prewarm: mean duration / main-thread pull / stopped early | 0.21 s /
22 ms / 0.1% | 0.10 s / 7 ms / 0% |
| Ledger close | 0.229 s | 0.164 s |

Notes:
- **No A/B on c6id.** #5458 has not been run on c6id, so the c6id PASS
shows this head can hold the target on current-generation x86. It
doesn't show how much of that comes from this PR. Every earlier run of
this fixture was on c5d.
- **Two regimes.** On c6id the network keeps up, so sets stay at about
6,100 transactions and triggers aren't all due at once; #5454's early
preparation runs, giving about 2 builds per ledger network-wide. On c5d
it falls behind, sets are full, and every ledger is heavier. So
per-ledger timings aren't like for like across the two.
- **What c5d still needs** (about 0.24 s to pass):
- Received-set validation still takes 0.33 s with 98.7% of its
signatures cached, so the rest of validation is now the cost.
  - About 245 ms of the c5d build is outside the phase timers.
- The remaining trigger-to-ballot time beyond build, fetch and
validation is about 0.57 s of nomination messaging across 4 regions.
- n = 1 per instance type.

---

## Previous update (2026-11-09)

_Genesis-upgrade fix, fetch robustness, follower signature prewarm._

Rebased onto the updated stack (#5453 now merges `master` @ 362dbc5:
protocol 30, millisecond close times; #5454, #5457 and #5458 updated the
same day). The PR's original four commits are kept (adapted to the new
base: stellar-xdr 30.0.0 paths, #5457's batched frame writes); ten
commits are added on top.

### Bug fix: networks starting at the genesis protocol stalled
(465f6a4)

**Root cause.** With leader-only construction (5fff543), followers no
longer build a tx set at
the trigger and must fetch the leader's. Before protocol 20 a ledger's
set is a legacy
`TransactionSet`, and the Rust overlay only carries
`GeneralizedTransactionSet`s (Core caches,
Rust decodes and Core receives only that type). So a follower could get
the leader's set only by
being promoted to leader by nomination timeouts and building it itself,
one slow round at a time.
Before this PR every validator built the identical empty legacy set (it
depends only on the
previous ledger; upgrades live in the `StellarValue`), so it was always
local. On perfnet (30 nodes)
22 nodes reached ledger 2 and stalled, 8 stayed at ledger 1. The upgrade
ledger itself
(LCL < 20) has the same problem; ledgers from protocol 20 on are
unaffected.

**Fix.** When the LCL is before protocol 20, every validator constructs
its nomination value at
the trigger again, as before; it is still voted only by round leaders.
Leader-only construction
is unchanged from protocol 20 on.

**Test.** `Rust overlay upgrades from genesis through legacy tx sets`: 4
nodes, genesis protocol 0,
four legacy ledgers at normal cadence, an upgrade straight to protocol
30, three more ledgers with
real (generalized) tx sets. It fails without the fix (a node stuck at
ledger 2) and passes with it.
Ablation on the final stack: disabling only this fix, with all the other
changes below in place,
still fails (2/2), so they don't mask the bug.

### Robustness on the same path
- 193fee3: a locally built set completes a pending fetch of the same
hash, so envelopes waiting
for it are processed at once (the IPC path had lost upstream's
ItemFetcher behaviour).
- 942a6b3: a follower promoted to leader after losing sync can still
propose; tracking is
checked at the trigger, as before. This is what made the perfnet stall
permanent.
- ab4eb14, 5964ce6: unanswered tx-set requests are retried and
reported. Core re-requests
undecided slots' sets with backoff (1, 2, 4, then every 8 s), warns on
every doubling of the
request count and counts `overlay.fetch.txset-retry`; Rust sends a retry
to a random other peer
and prunes stale reservations by slot. Core also checks a delivered
set's content hash.
- 5c159ae: test for a leader whose own build completes a pending fetch
and externalizes the
  slot inside SCP's nomination supplier.
- Rebase onto the new base: the test-only nomination override from #5481
now also hands its
substituted set to the overlay (31e6147), since this PR caches the
built set earlier.

### Performance
- 3903e4e, f2643ba: follower signature prewarm. The received-set
validation timer rose (about
158 -> 205 ms) because followers no longer validated their own mempool,
which used to warm the
signature-verification cache. Benchmark (6,000 one-signature txs, 4
tasks): 67 ms cold vs
10.5 ms warm. Followers now pull their top mempool transactions once at
the trigger and verify
master-key signatures on background tasks (at most 20 verifications per
tx; two worker threads
left free; skipped if the node already voted; stopped when validation, a
local build or
  externalization starts). They still build and compress nothing.
- c26b191: no early tx-set preparation when the trigger is already
due. Both timers fired at the
same instant in unspecified order, so a predicted leader could build its
set twice.
- 2ac0af9: no deep copies of tx-set XDR on the overlay IPC paths
(leader serializes in place,
follower decodes in place and moves into the frame). Sub-millisecond for
classic sets.

### Testing
- Laptop, on c26b191: new and changed tests (24, including the genesis
repro and the trigger,
  early-nomination and #5481 tests) and partitioned `make check` for

`[herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[nominationprotocol]~[.]`
  (143 test cases) plus the overlay's Rust tests (169 + 32 + 3) pass.
- Full suite `--all-versions` (devbox, at c26b191): selftest 598 test
cases / 8.5M assertions across protocol versions, check-sorobans,
check-rust and check-nondet pass.
- **Perfnet, head `c26b191fa`** (30 validators in 4 AWS regions, 10
tier-1 orgs, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply,
tx sets capped at 7,500; n = 1; run in parallel with the #5458 head on
the same fixture):
- **The genesis-upgrade bug is fixed on a real network:** all 30 nodes
synced in 43 s from genesis through the legacy ledgers to protocol 30
(the previous image of this PR left 22 nodes at ledger 2 and 8 at ledger
1).
- Correct under load: no forks, all in sync, no invalid
values/signatures, no restarts, no `NOT IN CACHE`/parse/cache-drop
errors; 11 tx-set fetch retries, all resolved on the first re-request.
- Leader-only construction works as designed (~1 build per ledger
network-wide instead of 30), and zstd shrinks each 7,500-tx SAC set from
~5.6 MB to ~1.0 MB (5.7x, ~7.5 ms to encode).
- Verdict FAIL at the same speed as the #5458 head (window-1 p75 3.26 s
vs 3.25 s): the leader's ~1.1 s build stays on the critical path, and
follower validation of the received set was 0.52 s vs 0.34 s on #5458
(whose followers had warmed their signature cache by building), so the
follower prewarm did not recover it at this load. Nomination round 1 (2
s timeout) times out on almost every ledger.
- Follow-up: done in the 2026-11-10 update above (leader build,
ballot-start prewarm, and the IPC size fix found by the next run).

---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet; the run of this
PR's final head will be posted in the update above.

# Description

This PR compresses transaction-set responses in the Rust overlay and
shares the encoded bytes across peer sends. Every response uses the
mandatory zstd protocol; there is no legacy/raw fallback or optional
lazy encoding state. The leader hands the final XDR to Rust before
Core's final validation, so compression can overlap that validation.
Receivers retain the original compressed bytes, decompress and verify
the set, then reuse the retained encoding when forwarding it.

A separate commit, `0ed3e540`, removes private proposals on non-leader
nodes. Followers enter nomination with a lazy value supplier and follow
received proposals without pulling the mempool or constructing a set.
First-round leaders can prepare early; a promoted leader constructs a
value only when it needs one. The driver caches that value across
retries, and externalization stops nomination even when no local value
was built. Tests cover timeout promotion, current close times, following
another leader, unavailable values, and reentrant nomination stop.

This PR builds on #5458. The
comparison below evaluates these revisions against the **original
compressed implementation of this PR**, using the same 10-tier-1,
30-validator fixture: 3,000 offered TPS, 6,000 transactions/block,
2-second target, 8 CPUs/16 GiB per validator, 60-second warmup and
900-second measurement, with simulated apply.

## 3,000 offered TPS

| Metric | Original compression | Updated leader-only path |
|---|---:|---:|
| Median block interval | 2.018 s | 2.120 s |
| Worst-node full-window p99 block interval | 3.834 s | 3.836 s |
| Tx-set fetch p99 | 0.617 s | 0.601 s |
| Completed TPS | 2,926.67 | 2,806.67 |
| Harness verdict | PASS | FAIL (ob-0 only) |

Median intervals and completed throughput use the canonical node;
worst-node p99 uses the fixed measurement window. The harness uses a
different histogram/window: `ob-0` recorded 4.989 s against its 4-second
limit, while the other 29 nodes and inclusion passed. Completed
throughput fell 4.1% and median block interval rose 5.1%; this run does
**not** establish that the revised path matches or improves the original
compression baseline. These are single cloud runs with normal host
placement.

Leader-only construction is verified: 886 builds/compressions across 421
closed ledgers, all correlated with asserted leadership, with no
follower build or received-set recompression detected. Multiple
nomination leaders can build in a ledger. Compared with the intermediate
eager implementation, local compressions fell from 13,019 to 893 in
their respective 900-second windows, roughly a 93% reduction per ledger.
Payload savings remain 80.0%. The sampled received-set validation timer
rose from about 158 ms to 205 ms versus that intermediate build; the
precise cause is not isolated by this run.

# Checklist

- [x] Reviewed the contributing document.
- [x] Kept leader-only proposal construction in its own commit on this
PR's branch.
- [x] Formatting checked with `make format` and `cargo fmt --check`.
- [x] Local build, Jenkins package/image build, and exact-head Clang/GCC
CI passed.
- [x] Rust suite: 194 passed, 3 ignored; the leader-only follow-up
leaves those validated Rust sources unchanged.
- [x] Core validation: 41 focused SCP/nomination/tx-set cases, 17
overlay IPC cases, and all 72 Herder cases covered. PostgreSQL-dependent
cases were rerun after configuring the local database fixture.
- [x] Performance comparison, runtime identity, resource checks and
sealed-artifact audit completed.
- [ ] Performance matches or improves the original compressed baseline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants