Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
312 changes: 193 additions & 119 deletions README.md

Large diffs are not rendered by default.

6 changes: 6 additions & 0 deletions docs/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,12 @@ overlay.recv.<X> | timer | received message <X>
overlay.send.<X> | meter | sent message <X>
overlay.timeout.idle | meter | idle peer timeout
overlay.timeout.straggler | meter | straggler peer timeout
overlay.txset-push.dropped | meter | tx-set frames dropped undecoded by the per-peer limit on frames the node did not ask for
overlay.txset-push.evicted | meter | tx sets from peers evicted by the limits on sets Core has not asked for
overlay.txset-push.hit | meter | Core tx-set requests answered at once from a set a peer had pushed
overlay.txset-push.received | meter | tx sets received from a peer without a request
overlay.txset-push.request-skipped | meter | peer requests not answered because the set was already pushed to that peer
overlay.txset-push.sent | meter | tx sets pushed to a peer by a round leader, without a request (one per peer)
process.action.queue | counter | number of items waiting in internal action-queue
process.action.overloaded | counter | 0-or-1 value indicating action-queue overloading
process.file.handles | counter | number of open file handles
Expand Down
1 change: 1 addition & 0 deletions docs/rust-overlay/ipc.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ A mempool pull of 2x the tx-set capacity plus a pending set's transactions
| 11 | `RequestTxSet` | `[hash:32][slot:u32 LE]` | Fetch TX set body by hash |
| 12 | `CacheTxSet` | `[hash:32][slot:u32 LE][txset_xdr]` | Tell overlay to cache a locally-built TX set |
| 13 | `RequestOverlayMetrics` | (empty) | Request metrics snapshot |
| 14 | `PushTxSet` | `[hash:32][slot:u32 LE]` | Send a cached TX set to every peer now (leader push) |

### Overlay → Core

Expand Down
12 changes: 12 additions & 0 deletions docs/rust-overlay/transport.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,18 @@ Streams open when a peer connects and reopen on demand after failure. Every
opening applies the route's priority before writing application data. A fetch
reuses the existing stream; it does not open a new stream per request.

Every connection starts with a congestion window of 8 MiB instead of Quinn's
default of about ten packets (`initial_congestion_window` in the vendored
`libp2p-quic` config, see
[`congestion-window.patch`](../../overlay/vendor/congestion-window.patch);
`STELLAR_OVERLAY_QUIC_INITIAL_CWND` overrides it in bytes, 0 = Quinn's
default). Quinn paces one window over 0.8 round trips and does not grow the
window while the sender is application limited, so with the default a tx set
of 0.6 MB took 1 to 1.3 round trips on top of the one-way delay on a warm
connection and 6 to 7 on a cold one; with 8 MiB it leaves in one flight (about
20 ms after the one-way delay at a 150 ms round trip). CUBIC still shrinks the
window on loss. This is a prototype setting for validators on good links.

Streams still share connection-level flow control, congestion control, CPU,
and link bandwidth. This separation removes dispatcher and stream-lock
serialization; it does not guarantee SCP latency under link saturation.
Expand Down
94 changes: 91 additions & 3 deletions docs/rust-overlay/txset-fetching.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
# TX set fetching

SCP envelopes reference transaction sets by hash. Rust prefetches their bodies
from peers and keeps them in the tx-set cache. Core receives a set only after
it establishes demand with `RequestTxSet`.
SCP envelopes reference transaction sets by hash. A round's leader pushes the
set it is about to propose to its peers; a node that does not hold a
referenced set requests its body from a peer. Either way Rust keeps the bodies
in the tx-set cache, and Core receives a set only after it establishes demand
with `RequestTxSet`.

## Wire formats and routes

Expand Down Expand Up @@ -57,6 +59,86 @@ borrow those encoded bytes directly, so relays never recompress received sets.
Encoding is complete before cache insertion; sending has no lazy initialization
or compression decision. Core continues receiving uncompressed XDR over IPC.

## Leader push

A round leader does not wait to be asked. For the set it is about to propose,
Core sends `PushTxSet [hash:32][slot:u32 LE]` right after the `CacheTxSet`
that carried the set: from inside the build when the node leads the current
round, after the build for a first-round leader's early (prepared) set that
was capacity limited, and again when a prepared set is taken at the trigger.
App looks the hash up in the cache (a miss is logged and ignored) and the
dispatcher writes the prepared encoding to the tx-set stream of every
connected peer that has not been sent this set. The proposal itself follows
on the control stream once Core has validated and persisted it.

The pushed frame is an ordinary tx-set frame that nobody requested. It
carries no hash and no slot: the receiver decompresses, strict-parses and
hashes it like a response and caches it under the hash it computed. Core is
not told. When an SCP statement names that hash, Core's `RequestTxSet` is a
cache hit (`overlay.txset-push.hit`). A pushed set whose hash no statement
names is never delivered to Core.

Push against request. If the proposal reaches a peer before the pushed set is
complete, the peer requests the set as usual. The leader does not answer that
request (`overlay.txset-push.request-skipped`) if it is the first one from
that peer for a set that is still being written to it, or was written less
than 500 ms ago, on the current connection: the bytes are already on the same
ordered, reliable stream, and the peer's reservation is cleared when they
land. Every other request is answered: a repeated one, one that arrives later
(a peer only asks when it does not hold the set, so it lost or dropped the
pushed copy), and one from a peer the set was not pushed to. A failed push
write answers a skipped request after all, and any connection to the peer
closing drops the record for that peer. (If a connection closes while a push
is still being written, that push is no longer recorded: a request that
crosses it gets a second copy, and if the write then fails, a request that
was skipped before the close is not answered and falls to Core's retry.)

A leader keeps at most one push in flight per peer (`TXSET_PUSH_BUSY` in the
log when a peer is left out). A peer that does not read its tx-set stream is
not sent further sets (it requests them), so it pins at most one pushed set
in the leader's memory; closing a connection does not release that count,
only the end of the write does. Answers to that peer's requests queue behind
the stuck write under the send permits that bound all tx-set responses.

A node that did not receive the push (not connected to the leader, connected
later, or busy with an earlier push) has no cached set and uses the request
path below. A push that the receiver itself dropped (rate limit) or evicted
right away (limits on unclaimed sets) is not recovered that fast: the
receiver's request reaches the leader within the 500 ms and is skipped, so the
set only comes with Core's retry to another peer after one second. The window
helps requests that arrive later (an eviction after some time, a first request
that went to a relay). The push is not relayed: a
node that is not connected to the leader requests the set from the peer that
relayed the proposal, which holds the pushed copy.

Limits on what a peer can make a node decode and keep:

- Rate: every tx-set frame takes a token from a per-peer bucket (burst 8, 4
per second). A frame that delivers a set this node has a request pending
for returns its token (to the peer that delivered it, whichever peer was
asked), so only frames nobody is waiting for use the allowance. With the bucket empty a
frame still passes if this node sent that peer a request it has not used up
(one frame per request, at most 8 saved). Anything else is dropped before
decompression (`overlay.txset-push.dropped`).
- Space: cached sets that came from a peer and that Core has not asked for
are limited per peer, oldest dropped first: 4 that the peer pushed and,
separately, 16 that answered this node's own requests (a peer answers a new
connection's SCP state request with about 12 slots at once). Together they
are limited to 128 MiB (the peer holding the most bytes loses its oldest
set). Evictions count as `overlay.txset-push.evicted`. A set leaves this
accounting when Core requests or builds it, or when the cache drops it. A
set is accounted to the peer that delivered it first.
- Not bounded by these: a peer can make a node send it requests, because the
overlay prefetches any hash named in an SCP envelope before Core checks the
signature; each such envelope buys one frame over the rate. A peer that
reconnects starts with a full allowance. Many colluding peers can use the
shared byte budget to push out a large honest set, and a peer that delivers
the leader's set first and then pushes four more evicts it (the receiver
then requests the set). The cache itself still holds 100 sets of any kind,
first in, first out: with up to 20 unclaimed sets per peer, five peers can
fill it and cycle out sets the node built or already uses (on the base one
peer could, through prefetches).

## Core requests and network prefetches

`RequestTxSet` IPC carries `[hash:32][slot:u32 LE]`. App records the newest
Expand Down Expand Up @@ -121,6 +203,12 @@ See [mempool](mempool.md).

## Remaining limitations

- The push is not relayed and carries the full compressed set; nodes that are
not connected to the leader, and a compact (hash-list) form, are future work.
- A peer whose first copy of a proposal came through a relay requests the set
from that relay, which answers from its cache while the leader's push may
still be in flight: one duplicate transfer.

- A cache miss at the selected peer has no explicit negative response; the
requester only notices through Core's one-second retry.
- Selecting one connected peer does not guarantee it has the requested body.
Expand Down
5 changes: 5 additions & 0 deletions overlay/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ pub struct Config {

/// Log level
pub log_level: String,

/// Directory in which TX sets are stored durably before they are handed
/// to Core (see `flood::TxSetStore`). No store if unset.
pub txset_dir: Option<PathBuf>,
}

impl Default for Config {
Expand All @@ -29,6 +33,7 @@ impl Default for Config {
libp2p_listen_ip: "0.0.0.0".to_string(), // Bind to all interfaces for internet operation
peer_port: 11625,
log_level: "info".to_string(),
txset_dir: None,
}
}
}
Expand Down
12 changes: 12 additions & 0 deletions overlay/src/flood/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ mod pending_requests;
mod tx_buffer;
mod txset;
pub(crate) mod txset_encoding;
pub(crate) mod txset_push;
mod txset_store;

pub use inv_batcher::InvBatcher;
pub use inv_messages::{GetData, InvBatch, InvEntry, TxStreamMessage, INV_BATCH_MAX_SIZE};
Expand All @@ -19,3 +21,13 @@ pub use pending_requests::PendingRequests;
pub use tx_buffer::TxBuffer;
pub use txset::{CachedTxSet, Hash256, TxSetCache};
pub use txset_encoding::TxSetData;
pub use txset_push::{PushRateLimiter, PushRecord, UnclaimedSetBudget};

/// Limits of the tx-set push, for tests outside this crate's library.
pub mod txset_push_limits {
pub use super::txset_push::{
MAX_UNCLAIMED_BYTES, MAX_UNCLAIMED_REQUESTED_SETS_PER_PEER, MAX_UNCLAIMED_SETS_PER_PEER,
PUSH_BURST, PUSH_REFILL_PER_SEC, PUSH_SKIP_WINDOW,
};
}
pub use txset_store::TxSetStore;
39 changes: 39 additions & 0 deletions overlay/src/flood/txset.rs
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,16 @@ impl TxSetCache {
self.by_hash.get(hash)
}

/// Remove one TX set. Returns true if it was cached.
pub fn remove(&mut self, hash: &Hash256) -> bool {
if self.by_hash.remove(hash).is_some() {
self.order.retain(|h| h != hash);
true
} else {
false
}
}

/// Remove TX sets for ledgers before the given sequence.
pub fn evict_before(&mut self, ledger_seq: u32) {
self.order.retain(|hash| {
Expand Down Expand Up @@ -121,6 +131,35 @@ mod tests {
assert!(cache.get(&[2u8; 32]).is_some()); // kept
}

#[test]
fn test_cache_remove_keeps_order_in_sync() {
let mut cache = TxSetCache::new(2);
for (seed, seq) in [(1u8, 100), (2u8, 101)] {
cache.insert(CachedTxSet {
hash: [seed; 32],
xdr: Arc::new(TxSetData::from_local(vec![]).unwrap()),
ledger_seq: seq,
});
}
assert!(cache.remove(&[1u8; 32]));
assert!(!cache.remove(&[1u8; 32]));
assert_eq!(cache.len(), 1);

// The removed hash no longer takes part in capacity eviction: adding
// two more sets evicts [2] and keeps the cache at capacity.
for (seed, seq) in [(3u8, 102), (4u8, 103)] {
cache.insert(CachedTxSet {
hash: [seed; 32],
xdr: Arc::new(TxSetData::from_local(vec![]).unwrap()),
ledger_seq: seq,
});
}
assert_eq!(cache.len(), 2);
assert!(cache.get(&[2u8; 32]).is_none());
assert!(cache.get(&[3u8; 32]).is_some());
assert!(cache.get(&[4u8; 32]).is_some());
}

#[test]
fn test_cache_capacity_eviction() {
let mut cache = TxSetCache::new(2); // Small cache
Expand Down
Loading
Loading