Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
4622ce4
Compress and share transaction set responses across peers
SirTyson Sep 15, 2026
a1060af
Prepare mandatory compressed transaction sets before validation
SirTyson Sep 17, 2026
e545aca
Clarify compressed transaction set delivery ownership
SirTyson Sep 17, 2026
223f00e
Build local transaction sets only for nomination leaders
SirTyson Sep 17, 2026
2b05be1
Build legacy tx sets on every validator under leader-only nomination
SirTyson Nov 9, 2026
8fcfe5d
Complete pending tx-set fetches with locally built sets
SirTyson Nov 9, 2026
e5d8948
Let promoted nomination leaders propose after losing sync
SirTyson Nov 9, 2026
9b6ef06
Retry stalled tx-set fetches and report sets that never arrive
SirTyson Nov 9, 2026
3630d31
Avoid copying tx-set XDR on the overlay IPC paths
SirTyson Nov 9, 2026
260c4a7
Pre-verify mempool signatures on followers at the trigger
SirTyson Nov 9, 2026
18fa41b
Back off tx-set fetch retries and retry only undecided slots
SirTyson Nov 9, 2026
2798f92
Bound the follower signature prewarm
SirTyson Nov 9, 2026
ba7ef4d
Test externalizing inside a leader's own tx-set build
SirTyson Nov 9, 2026
3c5d885
Skip early tx-set preparation when the trigger is already due
SirTyson Nov 9, 2026
11d416d
Time tx-set build phases and add a hidden leader build benchmark
SirTyson Nov 9, 2026
925dca9
Validate only as many tx-set candidates as selection can use
SirTyson Nov 9, 2026
1e16ae0
Hash tx-set candidates on the validation workers
SirTyson Nov 9, 2026
f199539
Precompute the parallel tx-set builder's sort keys
SirTyson Nov 9, 2026
d03d56f
Decode mempool pulls in place from the overlay IPC payload
SirTyson Nov 10, 2026
547e29f
Give Core's tx-set fetch metrics their own names
SirTyson Nov 10, 2026
18f84ac
Prewarm the next ledger's signatures from the ballot on every validator
SirTyson Nov 10, 2026
ab58eae
Raise the Core-overlay IPC payload limit to 64 MiB and never exceed it
Nov 10, 2026
1a52bf6
Run make format
SirTyson Nov 10, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

21 changes: 20 additions & 1 deletion docs/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,23 @@ crypto.verify.tx-valid-hit | meter | signature cache hits tha
crypto.verify.tx-valid-total | meter | sum of both hits and misses during transaction validation (outside of background signature validation)
herder.txset.build | timer | time spent building the tx set proposed at nomination from the transaction queues
herder.txset.validate | timer | time spent turning a received tx set into an applicable tx set and validating it on a validity-cache miss
herder.txset.validate-sig-cache-hit | meter | signature cache hits while validating a received tx set (herder.txset.validate)
herder.txset.validate-sig-cache-miss | meter | signatures verified (not in the cache) while validating a received tx set
herder.txset.build-pull | timer | tx set build phase: pulling and decoding the mempool's top transactions
herder.txset.build-frames | timer | tx set build phase: making transaction frames, splitting them into phases, one per source account
herder.txset.build-trim | timer | tx set build phase: validating candidates (each phase until it has enough valid ones)
herder.txset.build-select | timer | tx set build phase: surge pricing and building the parallel Soroban phase
herder.txset.build-xdr | timer | tx set build phase: building the set's XDR
herder.txset.build-handoff | timer | tx set build phase: handing the built set to the overlay
herder.txset.build-roundtrip | timer | tx set build phase: re-reading the built set from its XDR
herder.txset.build-final-check | timer | tx set build phase: final structural check of the built set
herder.txset.build-sig-cache-hit | meter | signature cache hits while a leader's build validates its candidates
herder.txset.build-sig-cache-miss | meter | signatures verified (not in the cache) while a leader's build validates its candidates
herder.prewarm.pull | timer | main thread time of a signature prewarm's mempool pull (the overlay round trip; decoding runs in the prewarm's tasks)
herder.prewarm.duration | timer | time from a signature prewarm's start until its background tasks finished (or it was stopped)
herder.prewarm.txs | meter | transactions whose signatures a signature prewarm checked
herder.prewarm.verified | meter | signatures a signature prewarm verified (were not in the cache yet)
herder.prewarm.incomplete | meter | signature prewarms stopped (by a build, a validation or a newer prewarm) before they finished
herder.pending[-soroban]-txs.age0 | counter | number of gen0 pending transactions
herder.pending[-soroban]-txs.age1 | counter | number of gen1 pending transactions
herder.pending[-soroban]-txs.age2 | counter | number of gen2 pending transactions
Expand Down Expand Up @@ -130,7 +147,7 @@ overlay.delay.async-write | timer | time between each messag
overlay.delay.write-queue | timer | time between each message's entry and exit from peer write queue
overlay.error.read | meter | error while receiving a message
overlay.error.write | meter | error while sending a message
overlay.fetch.txset | timer | time to complete fetching of a txset
overlay.fetch.txset | timer | time the Rust overlay took to fetch a txset from a peer (synced from the overlay as per-batch averages)
overlay.fetch.qset | timer | time to complete fetching of a qset
overlay.fetch.unique-recv | meter | number of bytes of fetched messages that have not yet been received
overlay.fetch.duplicate-recv | meter | number of bytes of fetched messages that have already been received
Expand Down Expand Up @@ -181,6 +198,8 @@ scp.envelope.receive | meter | SCP message received
scp.envelope.sign | meter | envelope signed
scp.envelope.validsig | meter | envelope signature verified
scp.fetch.envelope | timer | time to complete fetching of an envelope
scp.fetch.txset | timer | time from requesting a txset an SCP envelope needs to the overlay delivering it (not counting sets this node built itself)
scp.fetch.txset-retry | counter | txset requests repeated because the set had not arrived
scp.memory.cumulative-statements | counter | number of known SCP statements known
scp.nomination.combinecandidates | meter | number of candidates per call
scp.pending.discarded | counter | number of discarded envelopes
Expand Down
10 changes: 8 additions & 2 deletions docs/rust-overlay/ipc.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,14 @@ Source:
| `length` | 4 | Payload length in bytes. Native-endian. |
| `payload`| `length` | Type-specific bytes. May be empty. |

Maximum payload: **16 MB** (`messages.rs:154`). Frames larger than that
are rejected with `InvalidData`.
Maximum payload: **64 MiB** (`MAX_PAYLOAD_SIZE` in `messages.rs`, which must
equal `ipc::MAX_PAYLOAD_BYTES` in Core's `src/overlay/IPC.h`). Readers reject
larger frames (`InvalidData` in Rust; Core stops reading the stream), and
neither side sends one: `CoreSender::send` refuses it with
`IpcError::PayloadTooLarge`, Core's `sendMessage` returns false, and the
overlay truncates a top transactions response to the longest prefix that fits.
A mempool pull of 2x the tx-set capacity plus a pending set's transactions
(22,700 SAC payments, about 17.2 MB) exceeded the old 16 MiB limit.

> Native-endian is unusual for a wire protocol but safe here because
> Core and Overlay always run on the same host. The libp2p network
Expand Down
16 changes: 5 additions & 11 deletions docs/rust-overlay/transport.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,20 +74,15 @@ QUIC connection. Send priorities are set in Quinn, the QUIC implementation:
| Protocol ID | Messages | Send priority |
|-------------|----------|---------------|
| `/stellar/control/1.0.0` | SCP envelopes, `GET_SCP_STATE`, `GET_TX_SET` | **2 — highest** |
| `/stellar/txset/1.0.0` | Tx-set responses | **1** |
| `/stellar/txset/zstd/2.0.0` | Compressed tx-set responses | **1** |
| `/stellar/tx/1.0.0` | Transaction INV, GETDATA, TX | **0 — lowest** |

SCP and tx-set requests share one FIFO stream; there is no separate priority
between messages on that stream. `DONT_HAVE` handling is not implemented.

Older peers remain supported. An explicit rejection of the control protocol
causes the sender to negotiate `/stellar/scp/1.0.0` for SCP and SCP-state
requests, at priority 2. Tx-set requests then use a separate stream with the
legacy `/stellar/txset/1.0.0` protocol, also at priority 2. This fallback opens
on demand and is reused, preserving request isolation from responses while
using four outbound streams for old peers. Connection errors do not trigger
protocol downgrade. Updated receivers continue accepting both legacy
protocols, including tx-set requests on `/stellar/txset/1.0.0`.
These routes are mandatory on this experimental branch. There is no fallback
to the old SCP or transaction-set protocols. Transaction-set responses always
use zstd; see [TX set fetching](txset-fetching.md) for framing and cache ownership.

In addition, libp2p's `Identify` protocol runs as `/stellar/1.0.0`
(`libp2p_overlay.rs:380`). It exchanges peer-id + listen addresses on
Expand All @@ -105,8 +100,7 @@ as independent tasks, so:
- INV batches and TX response data flow on `/stellar/tx/1.0.0` only.
- Between updated peers, tx-set fetch requests share the control stream with
SCP, so a large response to the same peer cannot hold up a request behind
its frame or stream lock. With legacy peers, a separate request stream
preserves that isolation as described above.
its frame or stream lock.
- Loss recovery on one stream does not require another stream to wait for
that stream's missing bytes.

Expand Down
75 changes: 63 additions & 12 deletions docs/rust-overlay/txset-fetching.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,56 @@ it establishes demand with `RequestTxSet`.

## Wire formats and routes

Network messages use a four-byte big-endian frame length followed by a
`StellarMessage`. `GetTxSet` contains the 32-byte hash; `GeneralizedTxSet`
contains the set XDR. The response hash is computed from the canonical bytes
after strict decoding, without re-encoding the set.
Requests use a four-byte big-endian frame length followed by a
`StellarMessage::GetTxSet` containing the 32-byte hash. Responses require
`/stellar/txset/zstd/2.0.0` and contain an outer four-byte big-endian frame
length, a four-byte big-endian original XDR length, and exactly one zstd frame.
There is no raw encoding or legacy protocol fallback on this experimental branch.

The decoder bounds both encoded and original sizes, requires one complete zstd
frame and an exact output length, and strict-parses `GeneralizedTransactionSet`.
The response hash is computed from the recovered canonical XDR without
re-encoding. The XDR limit remains 16 MiB minus four bytes; the encoded-frame
limit also permits zstd's worst-case expansion for incompressible input.

Requests share the highest-priority control route with SCP. Responses use the
next-priority tx-set route. Legacy-only peers use separate request and response
streams under the old tx-set protocol, while SCP uses its old protocol. All
routes share QUIC connection limits. See [transport](transport.md).
next-priority tx-set route. All routes share QUIC connection limits.
See [transport](transport.md).

## Encoding ownership

Only nomination leaders construct local proposals. Followers start SCP with a
lazy value supplier and vote for received leader proposals without building a
private set or compressing it. At the trigger a follower pulls its mempool's top
transactions once and verifies their master-key signatures on background
threads, so that validating the leader's set hits the signature cache (cold, it
costs about six times as much). The prewarm stops when validation of a received
set or a local build starts. A node promoted by a
nomination timeout builds only if SCP needs its own value; it rechecks ledger
state and chooses a current close time then. Subsequent rounds reuse that value.
First-round leaders may prepare before the trigger; later leaders do no
speculative preparation. Nomination stops on externalization even for followers
that never constructed a local value.

Ledgers closing on a pre-protocol-20 LCL are the exception. Their legacy
`TransactionSet`s cannot be carried by this overlay, so every validator builds
its own value at the trigger, as before leader-only construction. With an empty
mempool (genesis and the upgrade-only ledgers before protocol 20) the set depends
only on the previous ledger, so each validator already holds the leader's set.

Core sends `CacheTxSet` as soon as a locally constructed proposal's final XDR is
available, before the builder's roundtrip and final validation. Rust eagerly
compresses it at zstd level 1 on a blocking worker while Core continues those
checks. App awaits the prepared representation before processing another IPC
message or application event; the network dispatcher remains independent.
Repeated local publication of an already cached hash reuses its encoding.

A receiving overlay retains the original encoded allocation while decompressing
and strict-parsing on a blocking worker. The cache shares an immutable
`Arc<TxSetData>` containing both original XDR and encoded bytes. Peer responses
borrow those encoded bytes directly, so relays never recompress received sets.
Encoding is complete before cache insertion; sending has no lazy initialization
or compression decision. Core continues receiving uncompressed XDR over IPC.

## Core requests and network prefetches

Expand Down Expand Up @@ -49,13 +90,21 @@ the attempt returns without reserving the hash.

A failed write removes only its own reservation. Disconnect cleanup removes
reservations assigned to that peer. A received response clears its matching
reservation and records fetch latency and the requested slot. This does not
provide a timeout or automatic retry for a silent peer.
reservation and records fetch latency and the requested slot. `LedgerClosed`
drops reservations older than the retained-slot window (`sequence - 12`).

Rust has no request timer. Core repeats a `RequestTxSet` for a set it has not
received after one second, then every second, logging a warning on every
doubling of the request count (and counting `scp.fetch.txset-retry`).
A request for a hash Core is still waiting for is a retry: it replaces the
reservation and goes to another connected peer, preferring the SCP source only
if it was not the previous target, else a random one. Core stops fetching when
the slots waiting for the set are purged.

## Cache and externalization

The cache holds up to 100 sets in App. Each entry contains its content hash,
canonical XDR, and ledger sequence. Capacity eviction uses insertion order;
canonical XDR, encoded response, and ledger sequence. Capacity eviction uses insertion order;
updating an existing hash does not move it. `LedgerClosed` evicts entries older
than `sequence - 12`, using saturating subtraction. Pending Core demand expires
against the same retained-slot boundary.
Expand All @@ -72,6 +121,8 @@ See [mempool](mempool.md).

## Remaining limitations

- A peer that stays connected but silent can leave a fetch pending indefinitely.
- A cache miss at the selected peer has no explicit negative response or retry.
- A cache miss at the selected peer has no explicit negative response; the
requester only notices through Core's one-second retry.
- Selecting one connected peer does not guarantee it has the requested body.
- Legacy (pre-protocol 20) `TransactionSet`s cannot be fetched at all; see
encoding ownership.
3 changes: 3 additions & 0 deletions overlay/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ libp2p = { version = "0.54", features = ["tokio", "macros", "identify", "quic"]
libp2p-stream = "0.2.0-alpha"
futures = "0.3"

# TX set response compression
zstd = { version = "0.13.3", default-features = false }

# Logging
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
Expand Down
2 changes: 2 additions & 0 deletions overlay/src/flood/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ mod mempool;
mod pending_requests;
mod tx_buffer;
mod txset;
pub(crate) mod txset_encoding;

pub use inv_batcher::InvBatcher;
pub use inv_messages::{GetData, InvBatch, InvEntry, TxStreamMessage, INV_BATCH_MAX_SIZE};
Expand All @@ -17,3 +18,4 @@ pub use mempool::Mempool;
pub use pending_requests::PendingRequests;
pub use tx_buffer::TxBuffer;
pub use txset::{CachedTxSet, Hash256, TxSetCache};
pub use txset_encoding::TxSetData;
Loading
Loading