diff --git a/README.md b/README.md
index d6f248438..7973a937c 100644
--- a/README.md
+++ b/README.md
@@ -1,20 +1,28 @@
-# stellar-core: leader schedule, leader tx-set push and apply-time tx-set validation (experimental branch)
+# stellar-core: leader schedule with terms, leader tx-set push, apply-time tx-set validation, transactions sent to the leaders only (experimental branch)
**This branch is an experiment. It is not production code and must not run on a real network.**
The leader schedule and the tx-set push have no feature flag and no protocol gate, and the branch
is not compatible with nodes that nominate.
-The benchmark is now **48 validators (16 organizations), 5,000 TPS, 1,000 ms target block time**.
-The latest experiment (vote without validating the tx set, validate and execute at apply) was
-measured there on commit `0c9a02e56`, the code of this branch, on 2026-11-11. "Before" is the
-previous experiment on this branch (the leader's tx-set push), commit `8f7768be7`, measured on the
-same day on the same fixture. One run per build.
+The latest two experiments (leader terms with pre-validation, and transactions sent to the
+upcoming leaders only) were measured on 2026-11-12 at an **800 ms target block time on 16-core
+instances**: at 48 validators (16 organizations) and 8,000 TPS, and at the goal of this work,
+**90 validators (30 organizations) and 10,000 TPS**. One run per build.
+
+- **48 validators, 8,000 TPS:** with both experiments every window is on target (block time p75
+ 804-805 ms) and a node's overlay traffic is 6.2 MB/s. Without them the branch drifts and misses
+ its second window (846, then 869 ms) at 21 MB/s.
+- **90 validators, 10,000 TPS:** not reached yet. With both experiments the first window is at 980
+ ms (limit 960), a ledger every 0.90 s, 15.7 MB/s per node; with leader terms and pre-validation
+ alone it is at 1,032 ms, 1.02 s per ledger and 43 MB/s.
+
+The earlier benchmark (48 validators, 5,000 TPS, 1,000 ms, 4-core instances) is kept below.
## What this branch is
It is the overlay-v2 stack up to [#5459](https://github.com/stellar/stellar-core/pull/5459)
(Rust overlay over QUIC, millisecond close times, compressed tx-set responses, leader-only tx-set
-construction) plus three experiments:
+construction) plus five experiments:
1. **No nomination** ([#5495](https://github.com/stellar/stellar-core/pull/5495)). The SCP
nomination phase is deleted and replaced by a leader schedule that every node computes locally.
@@ -58,13 +66,156 @@ construction) plus three experiments:
- Details, safety argument and review findings:
[#5498](https://github.com/stellar/stellar-core/pull/5498).
-Unchanged by all three: the ballot protocol's accept, confirm and commit rules. For a valid tx
+4. **Leader terms, pre-validation during the previous ballot, one build before the trigger**
+ ([#5502](https://github.com/stellar/stellar-core/pull/5502)). No protocol gate, like the
+ schedule it changes.
+ - **Terms.** The node whose value closed a ledger leads the next one, for up to 5 ledgers; a
+ term ends on a ledger number divisible by 5. The order of a slot is seeded with the hash of
+ the ledger before the last closed one, so it is known one ledger earlier. A leader loses its
+ term when its round times out or its ledger applies nothing.
+ - **Pre-validation.** The nodes that expect to lead the next ledger check the validity of their
+ mempool's top candidates against the state the speculative execution left, on background
+ threads, while the executed ledger is still in its ballot. The results are used only if the
+ ledger closes with the value that was executed.
+ - **One build.** The first-round leader builds once, a lead time before its trigger, pushes
+ the set as soon as it exists and proposes it unchanged at the trigger.
+5. **Transactions go to the upcoming leaders only**
+ ([#5501](https://github.com/stellar/stellar-core/pull/5501), the latest experiment). No
+ protocol gate and no version negotiation.
+ - **Targets.** A node that accepts a transaction no longer advertises it to every peer. Its
+ overlay sends the body straight to the leaders of the node's current round and of the next
+ one for the next ledger and, one ledger before a new leader term starts, to that term's
+ first leader. One function in Core decides the list (`HerderImpl::computeTxFloodTargets`).
+ - **Keep and follow.** The accepting node keeps the transaction until it is externalized. When
+ the targets change it goes to a new target only while it is fresh (under 200 ms) or once
+ three ledgers closed without it. If a target is not a directly connected, identified peer,
+ the node floods as before.
+ - **Admission check.** A submitted transaction is checked against the last closed ledger
+ before it is sent on, with one pending transaction per source account. Before, a node
+ flooded whatever it was given.
+ - **Identity.** An overlay proves which validator it belongs to with a binding of its
+ connection key, signed by the node key.
+ - **Cost.** Followers no longer hold the transactions before the leader's set arrives, so
+ they verify signatures when they check the set at apply instead of in the background.
+
+Unchanged by all five: the ballot protocol's accept, confirm and commit rules. For a valid tx
set, ledger state and meta are also unchanged; experiment 3 changes when a set is validated and
-executed, and what an invalid set does to the ledger.
+executed, and what an invalid set does to the ledger; experiment 4 changes who leads; experiment
+5 changes which transactions reach a leader, and which submissions a node refuses.
## Performance results
-Target block time 1,000 ms, 5,000 TPS, 48 validators in 4 regions.
+### 800 ms target on 16-core instances: 48 validators, 8,000 TPS
+
+c6id.8xlarge (16 cores), 4 regions, SAC payments, 150 ms simulated apply, 900 s of load; one run
+per build, all on 2026-11-12. "Base" is this branch up to experiment 3.
+
+| Windows 1 / 2 / 3 / 4 | Base (`0c9a02e56`) | + leader terms and pre-validation (#5502, `deec2d87c`) | + transactions to leaders only (#5501, `6b3c7ae0e`) |
+|---|---:|---:|---:|
+| Block time windows on target | window 1 only; window 2 missed, the load stops there | **all four**, on all 48 nodes | **all four**, on all 48 nodes |
+| p75, average over 48 nodes (limit 960 ms) | 846 / 869 ms, rising | 807 / 806 / 805 / 805 ms | **805 / 804 / 804 / 804 ms** |
+| p99, average over 48 nodes (limit 1,600 ms) | 1,368 / 1,480 ms | 1,153 / 1,188 / 1,100 / 1,249 ms | 1,170 / 1,209 / 1,039 / 1,172 ms |
+| Time per ledger | 0.84-0.85 s | 0.81 s | 0.81 s |
+| Leader builds per ledger; build time | 1.9; 0.16 s | 1.0; 0.16 s, before the trigger | 1.0; 0.15 s, before the trigger |
+| Trigger to first ballot | 0.34-0.36 s | 0.11 s | 0.10-0.12 s |
+| First ballot to externalize | 0.29 s | 0.29 s | 0.29 s |
+| Externalize to ledger closed | 0.04 s | 0.05 s | 0.04 s |
+| **Trigger to ledger closed** | 0.67-0.69 s | **0.44-0.45 s** | **0.43-0.44 s** |
+| Tx-set validation at apply | 0.08 s | 0.08 s | 0.10 s (signatures verified inline) |
+| **Overlay traffic per node, each way** | 21.0 MB/s | 21.2 MB/s | **6.2 MB/s** |
+| Overlay messages received per node | 20,000 /s | 20,500 /s | 12,550 /s |
+| Transaction adverts sent per node | 7,848 /s | 7,848 /s | 0 |
+| Leader and ballot timeouts, empty tx sets | 0 | 0 | 0 |
+| Run verdict | FAIL (block time) | FAIL (inclusion check only) | FAIL (inclusion check only) |
+
+- **Leader terms and the single early build take the build off the critical path**: trigger to
+ ledger closed falls from 0.68 to 0.44 s, and the branch holds 800 ms at 8,000 TPS for the full
+ 15 minutes instead of drifting.
+- **Sending transactions to the leaders only removes 71% of the overlay traffic** at the same
+ block time. Both runs sit at the floor of this fixture (the stages fill 0.44 s of the 0.8 s
+ cycle), so this fixture cannot show a block-time difference between them.
+- What is left of the traffic is SCP (12,100 messages per second per node), the leader's pushed
+ set (about 5 MB per ledger) and 3.3 transaction bodies per transaction. The build measured
+ here re-sent every pending transaction when the targets changed; the head of #5501 no longer
+ does (2.5 bodies per transaction in the next table's run).
+- **Inclusion.** The check is an exact comparison (see [The inclusion check](#the-inclusion-check)).
+ #5502 alone held 80 transactions more than the 7,680,000 offered, the run with #5501's first
+ version 6,610 more (0.086%): copies of transactions that reach a node after their ledger
+ closed, included a second time, which only the test's "overlay-only" mode permits. With the
+ head's rule for re-sends the next run had 81 on 3,870,249.
+
+### 800 ms target at the goal: 90 validators, 10,000 TPS
+
+30 organizations, 22-23 validators per region, the same instances and load shape, 10,080 TPS
+offered. Both runs missed their first window and the load was stopped after it.
+
+| Window 1 | Leader terms and pre-validation (#5502, `8aa7f6bf0`) | + transactions to leaders only (#5501, `6ab837c27`) |
+|---|---:|---:|
+| p75, average over 90 nodes (limit 960 ms) | 1,032 ms | **980 ms** |
+| p99, average over 90 nodes (limit 1,600 ms) | 1,887 ms (over) | **1,441 ms** (inside) |
+| Time per ledger | 1.02 s | **0.90 s** (median 0.88 s) |
+| The first minute of the load | 802 ms for 30 s, then a backlog | 805 ms over the whole minute |
+| Tx sets at the 10,000 cap | 93% | 39% |
+| Leader builds finished before the trigger | 0 of 40 sampled | 40 of 55 sampled |
+| Leader build; candidates in the mempool | 0.33 s; 20,200 | 0.24 s; 8,350 (median) |
+| Trigger to first ballot | 0.58 s | **0.39 s** |
+| First ballot to externalize | 0.32 s | 0.32 s |
+| Externalize to ledger closed | 0.07 s | 0.07 s |
+| **Trigger to ledger closed** | 0.97 s | **0.77 s** |
+| Tx set at a follower after the proposal | 0.23 s | 0.17 s |
+| Tx-set validation at apply | (not listed) | 0.14 s |
+| **Overlay traffic per node, each way** | 43 MB/s | **15.7 MB/s** |
+| Leader and ballot timeouts, empty tx sets | 0 | 0 |
+| Submitted against included at the end | 49,855 short (backlog) | 81 more (the backlog drained) |
+| Run verdict | FAIL (block time: p75 and p99) | FAIL (block time: p75) |
+
+- **Neither build holds 800 ms at 90 validators and 10,000 TPS.** With transactions sent to the
+ leaders only the miss is 20 ms on the window's p75 instead of 72 ms, p99 stays inside its
+ limit, and the network runs 12% faster (0.90 s against 1.02 s per ledger).
+- The gain is in trigger to first ballot: the leader's mempool holds about one ledger of
+ candidates instead of a backlog of 20,200, so its build is shorter and three of four builds
+ finish before the trigger again.
+- **Overlay traffic is 15.7 MB/s per node instead of 43.** At 90 validators most of what is left
+ is SCP: 35,700 messages per second per node.
+- There is no run of the base (up to experiment 3) at this fixture. In the #5502 column, the
+ set's arrival, the build and the traffic are from that PR's first run at this fixture
+ (`deec2d87c`, 1,046 ms); its rerun with `8aa7f6bf0` reported the same stage table.
+
+### Where a ledger's time goes at 90 validators and 10,000 TPS
+
+From the logs of twelve nodes (three per region) in the run with all five experiments, 56
+ledgers led by one of them:
+
+| Step on the leader | Mean | |
+|---|---:|---|
+| Previous ledger closed to proposal | 0.29 s | The build starts when the previous ledger closes (it needs that ledger) and takes 0.24 s for 8,500 transactions; the set is pushed while the build finishes |
+| Proposal to consensus | 0.53 s | Of which, on a follower: 0.17 s until the set is in hand, then 0.34-0.36 s of ballot |
+| Consensus to ledger closed | 0.07 s | The commit of the speculatively executed ledger |
+| **Sum** | **0.89 s** | Target 0.80 s. 41 of the 56 cycles took longer than 0.8 s |
+
+1. **The build is on the critical path again: 0.29 s.** It cannot start before the previous
+ ledger has closed, and the cycle has no slack left for it, so the trigger is already due when
+ it ends (the proposal went out within 20 ms of the build's end on half of the ledgers that
+ had an early build).
+2. **The leader's set takes 0.19 s to reach a follower's overlay**, 0.13 s of it even within the
+ leader's own region: the leader sends the full set (5.7 MB of XDR on average, compressed
+ once with zstd) to each of its 89 peers, up to 506 MB per ledger before compression. A
+ follower cannot vote before it has the set. Whether the sender's CPU or its network limits
+ this was not measured.
+3. **The ballot is 0.34-0.36 s** from the set in hand to consensus. It is message delay between
+ four regions: proposal to consensus is 0.43 s when the leader is in us-east-1, 0.58 s from
+ us-west-2 and 0.60 s from ap-southeast-1, and a ledger led from ap-southeast-1 takes 0.96 s
+ against 0.83 s from us-east-1.
+4. **A follower's execution ends just before consensus**: 0.15 s of validation (signatures
+ verified inline) plus the simulated apply, 0.31 s after the set arrived.
+5. The main thread is not saturated (a posted task waits 14 ms on average).
+
+### Earlier benchmark: 1,000 ms target, 5,000 TPS, 48 validators, 4-core instances
+
+The measurement of experiment 3 (vote without validating the tx set, validate and execute at
+apply), on commit `0c9a02e56` on 2026-11-11. "Before" is the previous experiment on this branch
+(the leader's tx-set push), commit `8f7768be7`, measured on the same day on the same fixture.
+One run per build.
| | Before: tx-set push (`8f7768be7`) | With apply-time validation (`0c9a02e56`) |
|---|---|---|
@@ -219,9 +370,9 @@ generators had submitted exactly 4,800,000 and none was rejected.
interrupted reads it survived, and the "before" build, with the unfixed reader, had no such
event in its 30 minutes either. Whether the change makes the event more likely is not known.
-### Where a ledger's time goes now
+### Where a ledger's time went at 1,000 ms and 5,000 TPS
-What is left of the 0.76-0.84 s from trigger to ledger closed:
+What was left of the 0.76-0.84 s from trigger to ledger closed after experiment 3:
1. **Trigger to first ballot: 0.41-0.47 s.** Most of it is the leader's build (0.24-0.25 s), on
the critical path. The first-round leader also builds once before the trigger and discards
@@ -258,6 +409,26 @@ The first two experiments were measured on a smaller fixture (10 organizations,
## How it was measured
+The sections below describe the earlier benchmark (1,000 ms, 5,000 TPS). The 800 ms runs of
+2026-11-12 differ from it in these points only:
+
+| | 48 validators, 8,000 TPS | 90 validators, 10,000 TPS |
+|---|---|---|
+| Instance type | c6id.8xlarge: 32 vCPU, 16 cores, the same processor | the same |
+| Organizations; validators per region | 16; 12 | 30; 22 or 23 |
+| Connectivity | full mesh, 47 peers | full mesh, 89 peers |
+| Target ledger close time | 800 ms | 800 ms |
+| Offered load | 8,000 TPS, about 167 per validator | 10,080 TPS, 112 per validator |
+| Maximum transactions per ledger | 8,000 | 10,000 |
+| Simulated apply, per operation | 23 us (150 ms per ledger at the offered load) | 18 us |
+| Window limits | p75 in [640, 960) ms, p99 at most 1,600 ms | the same |
+| Validation tasks (one per core but one) | 15 | 15 |
+| Logs | not saved; twelve nodes were read while the network was up | the same |
+| Builds | `0c9a02e56` (base), `deec2d87c` (#5502), `6b3c7ae0e` (#5501 on #5502) | `8aa7f6bf0` (#5502), `6ab837c27` (#5501 on #5502) |
+
+`--block-time-ms 800 --soroban-tps 8000` (or `--org-count 30 --soroban-tps 10000`),
+`--instance-type c6id.8xlarge` and no `--save-logs`; everything else as in the command below.
+
All runs use the `block_time_check` mission of perfnet, SDF's tooling for
performance networks on AWS. Each run launches a fresh network, brings it to the target settings,
runs the load, judges it, and terminates the network.
@@ -379,6 +550,28 @@ Defaults that matter: validators as generators, `--scp-timeout-ms 2000`,
## Known limitations of the prototype
+Transactions to the leaders only:
+- No protocol gate and no version negotiation: a node without the change ignores the identity
+ stream, and nodes with the change flood toward it.
+- A node whose targets are not all directly connected, identified peers floods its transactions
+ as before, so a sparse topology gains nothing.
+- The first leader of a new term is predicted one ledger ahead. If the ledger in between costs
+ that node its turn, another node leads and gets the pending transactions only at the close.
+- After a leader failure the next round's leader lacks the transactions that are older than
+ 200 ms until that ledger closes; they reach it after it.
+- A restarted node forgets the transactions it accepted and had not seen included. Before, every
+ node held them.
+- A peer can fill a leader's mempool with invalid transactions that outbid valid ones; the leader
+ drops them when it builds. A refused submission returns an error without its result code.
+- Followers verify the set's signatures at apply (0.10 s at 6,400 transactions, 0.14 s at 8,500,
+ on 15 tasks). On 4-core instances that cost made the change miss a 1,000 ms target at 5,000 TPS
+ unless validation also used the hyper-threads (`BATCH_EXECUTOR_TASKS`).
+- Copies of a transaction that reach a node after its ledger closed are counted
+ (`overlay.flood.tx-late-copy`), not dropped.
+
+Leader terms and pre-validation: see [#5502](https://github.com/stellar/stellar-core/pull/5502).
+The schedule change has no protocol gate, and a term gives one node five ledgers in a row.
+
Apply-time validation and speculative execution:
- A nondeterministic validity check used to cost liveness at most. It now forks the ledger. The
check's node-local inputs are the binary's protocol version, the network passphrase and, in test
@@ -424,7 +617,10 @@ Leader schedule:
Unchanged from stellar-core: see [INSTALL.md](./INSTALL.md), [CONTRIBUTING.md](./CONTRIBUTING.md)
and [`./docs`](./docs).
-Test status of this branch's code (`0c9a02e56`, the same tree as the merge of #5498):
+Test status with experiments 4 and 5 (`6ab837c27`, #5501 on top of #5502): the full test suite
+passes, also with `--all-versions`, including the Rust overlay's tests.
+
+Test status of the branch up to experiment 3 (`0c9a02e56`, the same tree as the merge of #5498):
- The full test suite passes (615 test cases).
- With `--all-versions` the suite passes on `a33da62c4`, the commit before the IPC fix. On
`0c9a02e56` one test case of the leader-schedule change failed ("PendingEnvelopes
diff --git a/docs/metrics.md b/docs/metrics.md
index 0f290a46f..0b2d839a9 100644
--- a/docs/metrics.md
+++ b/docs/metrics.md
@@ -62,6 +62,9 @@ crypto.verify.miss | meter | number of signature cach
crypto.verify.total | meter | sum of both hits and misses
crypto.verify.tx-valid-hit | meter | signature cache hits that occurred while validating transactions (outside of background signature validation)
crypto.verify.tx-valid-total | meter | sum of both hits and misses during transaction validation (outside of background signature validation)
+herder.tx-admission.account-busy | meter | submitted transactions refused because their source account already has a pending transaction accepted by this node
+herder.tx-admission.check | timer | time spent checking a submitted transaction against the last closed ledger before it is sent on
+herder.tx-admission.invalid | meter | submitted transactions refused because they are invalid against the last closed ledger
herder.txset.build | timer | time spent building the tx set proposed at nomination from the transaction queues
herder.txset.validate | timer | time spent turning a received tx set into an applicable tx set and validating it on a validity-cache miss
herder.txset.validate-sig-cache-hit | meter | signature cache hits while validating a received tx set (herder.txset.validate)
@@ -166,6 +169,11 @@ overlay.flood.broadcast | meter | message sent as broadcas
overlay.flood.duplicate_recv | meter | number of bytes of flooded messages that have already been received
overlay.flood.unique_recv | meter | number of bytes of flooded messages that have not yet been received
overlay.flood.tx-batch-size | histogram | number of transactions in a batch
+overlay.flood.tx-direct-recv | meter | transaction bodies received without a demand (sent directly by a peer), kept and not relayed
+overlay.flood.tx-late-copy | meter | transactions admitted although one of the last few externalized ledgers already has them (a copy that arrived late)
+overlay.flood.tx-direct-sent | meter | transaction bodies sent straight to a target leader, without an advert
+overlay.flood.tx-direct-sent-bytes | meter | bytes of those bodies
+overlay.flood.tx-local-flooded | meter | own transactions advertised to every peer because no target list was set or a target was not directly connected
overlay.inbound.attempt | meter | inbound connection attempted (accepted on socket)
overlay.inbound.drop | meter | inbound connection dropped
overlay.inbound.establish | meter | inbound connection established (added to pending)
@@ -179,6 +187,8 @@ overlay.message.broadcast | meter | message broadcasted
overlay.message.read | meter | message received
overlay.message.write | meter | message sent
overlay.message.drop | meter | message dropped due to load-shedding
+overlay.node-id.identified | counter | connected peers that proved which validator they belong to
+overlay.node-id.rejected | meter | validator identity claims from peers with a signature that does not bind them to the connection
overlay.outbound.attempt | meter | outbound connection attempted (socket opened)
overlay.outbound.cancel | meter | outbound connection cancelled
overlay.outbound.drop | meter | outbound connection dropped
diff --git a/docs/stellar-core_example.cfg b/docs/stellar-core_example.cfg
index c53968c5f..a25807f38 100644
--- a/docs/stellar-core_example.cfg
+++ b/docs/stellar-core_example.cfg
@@ -291,6 +291,30 @@ EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD = false
# also enabled.
BACKGROUND_TX_SIG_VERIFICATION = true
+# TX_FLOOD_TARGET_LEADERS (integer) default 2
+# Experimental. How many upcoming round leaders get the transactions this
+# node accepts (the leader of the node's current round of the next ledger and
+# the ones after it; one ledger before a new leader term starts, also that
+# term's expected first leader). The node sends them the transactions
+# directly instead of advertising them to every peer. 0 = every validator of
+# the leader schedule (no adverts either, but every validator holds every
+# transaction before the leader's set arrives).
+TX_FLOOD_TARGET_LEADERS = 2
+
+# TX_FLOOD_LEGACY_ADVERTS (boolean) default false
+# Experimental. If true, no target list is used: transactions are advertised
+# to every peer and fetched on demand (the behaviour before
+# TX_FLOOD_TARGET_LEADERS existed).
+TX_FLOOD_LEGACY_ADVERTS = false
+
+# BATCH_EXECUTOR_TASKS (integer) default 0
+# Experimental. Number of tasks (worker threads) used for CPU-bound batch
+# work: transaction validation when building and when applying a transaction
+# set, and the background signature verification. 0 = one per physical core
+# minus one. A larger value uses hyper-thread siblings once the physical
+# cores are taken.
+BATCH_EXECUTOR_TASKS = 0
+
# NTP_DRIFT_CHECK_SERVER (string) default "pool.ntp.org"
# Hostname of an NTP server that the node periodically queries (about every ten
# minutes) to detect and warn about drift of the local clock. This is detection
diff --git a/overlay/src/flood/mod.rs b/overlay/src/flood/mod.rs
index cdf067e88..6bce126a8 100644
--- a/overlay/src/flood/mod.rs
+++ b/overlay/src/flood/mod.rs
@@ -8,6 +8,7 @@ mod inv_tracker;
mod mempool;
mod pending_requests;
mod tx_buffer;
+pub(crate) mod tx_router;
mod txset;
pub(crate) mod txset_encoding;
pub(crate) mod txset_push;
@@ -19,6 +20,7 @@ pub use inv_tracker::InvTracker;
pub use mempool::Mempool;
pub use pending_requests::PendingRequests;
pub use tx_buffer::TxBuffer;
+pub use tx_router::{NodeId, SendPlan, TxRouter};
pub use txset::{CachedTxSet, Hash256, TxSetCache};
pub use txset_encoding::TxSetData;
pub use txset_push::{PushRateLimiter, PushRecord, UnclaimedSetBudget};
diff --git a/overlay/src/flood/tx_router.rs b/overlay/src/flood/tx_router.rs
new file mode 100644
index 000000000..494b4f21f
--- /dev/null
+++ b/overlay/src/flood/tx_router.rs
@@ -0,0 +1,946 @@
+//! Decides where this node's transactions go.
+//!
+//! Core knows the leader schedule and tells the overlay which validators get
+//! transactions now (the "targets", usually the leaders of the current and
+//! the next round), or that there is no such list ("everyone": the INV/GETDATA
+//! flood to all peers). The router keeps the transactions this node accepted
+//! from its own Core until they are externalized, removed or expire, and
+//! works out what has to be sent when one is added, when the targets change
+//! and when a peer proves which validator it is.
+//!
+//! Transactions go directly (the body, without an advert) only when every
+//! target is this node itself or a connected peer that identified itself.
+//! Otherwise the node floods them as before, so that they reach every mempool
+//! whatever the topology.
+//!
+//! Pure bookkeeping: no I/O, no clock of its own. The owner holds it in a
+//! mutex that is never held across an await.
+
+use std::collections::{HashMap, VecDeque};
+use std::hash::Hash;
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+
+use super::mempool::TxHash;
+use crate::wire::ValidatedTx;
+
+/// A validator's public key (the NodeID's ed25519 key).
+pub type NodeId = [u8; 32];
+
+/// At most this many accepted transactions are kept for sending again; the
+/// oldest is dropped first. Same bound as the mempool.
+pub const MAX_LOCAL_TXS: usize = 100_000;
+
+/// An accepted transaction is given up after this long. Same age as the
+/// mempool's.
+pub const LOCAL_TX_MAX_AGE: Duration = Duration::from_secs(300);
+
+/// A transaction is not sent again to a peer that got it less than this long
+/// ago on the same connection: a target that stays or soon is a target again
+/// still has it. Sending again after that covers a body the peer dropped
+/// under load (it has no way to ask for it), at a bounded cost while a
+/// backlog keeps many transactions pending.
+pub const RESEND_INTERVAL: Duration = Duration::from_secs(60);
+
+/// When the targets change, a kept transaction that already went to its
+/// targets is sent to a validator that newly became one only if it may still
+/// be wanted there:
+/// * it is younger than [`FOLLOW_TARGETS_MAX_AGE`], so it may have missed
+/// the set the previous targets were building when it reached them, or
+/// * [`FOLLOW_TARGETS_AFTER_LEDGERS`] ledgers externalized here since it
+/// was accepted and none of them had it. Three, counted on this node: the
+/// first may have been built before the transaction was accepted, the
+/// second before it reached the leader (this node may see ledgers late,
+/// and the transaction takes a network hop), the third was not. Or
+/// * it is older than [`FOLLOW_TARGETS_ANYWAY_AFTER`]: no ledger closes
+/// (rounds are timing out), and the leaders of later rounds need it.
+///
+/// Anything else is most likely in the set that is in consensus right now.
+/// Sending it on would put a copy into the new target's mempool that, on a
+/// node that is running late, arrives after that set externalized there:
+/// the new target then holds a finalized transaction and offers it again.
+///
+/// The age bound has to stay well below the time from a leader's build to
+/// the externalize of its set (about 0.5 s at 800 ms ledgers): a fresh
+/// transaction that did make the set must reach the new target before that.
+pub const FOLLOW_TARGETS_MAX_AGE: Duration = Duration::from_millis(200);
+pub const FOLLOW_TARGETS_AFTER_LEDGERS: u64 = 3;
+pub const FOLLOW_TARGETS_ANYWAY_AFTER: Duration = Duration::from_secs(3);
+
+/// Hashes of recently externalized transactions that are remembered, to
+/// count copies that arrive after their transaction is final (see
+/// [`TxRouter::was_finalized`]): about six ledgers of 10,000.
+pub const MAX_FINALIZED_TXS: usize = 60_000;
+
+/// Bodies remembered as "received directly from a peer" (see
+/// [`TxRouter::note_direct`]). A target receives every node's transactions,
+/// so this is about 20 s of a 5,000 TPS load; they are dropped earlier when
+/// they are externalized.
+pub const MAX_DIRECT_TXS: usize = 100_000;
+
+/// What to send, as decided by the router. The caller does the I/O.
+#[derive(Debug)]
+pub struct SendPlan
{
+ /// Bodies to send straight to a peer.
+ pub direct: Vec<(P, Vec>)>,
+ /// Transactions to advertise to every peer (the INV/GETDATA flood).
+ pub flood: Vec>,
+}
+
+impl
{
+ pub fn is_empty(&self) -> bool {
+ self.direct.is_empty() && self.flood.is_empty()
+ }
+}
+
+/// A transaction that arrived as a body nobody here asked for.
+struct DirectTx {
+ tx: Arc,
+ /// Advertised to this node's peers since (it joined a flood).
+ relayed: bool,
+}
+
+struct LocalTx
{
+ tx: Arc,
+ added: Instant,
+ /// The router's ledger count when it was accepted.
+ added_at_ledger: u64,
+ /// A plan has looked at it under a direct route: from then on it only
+ /// follows the targets under the rule of [`FOLLOW_TARGETS_MAX_AGE`].
+ planned: bool,
+ /// A send was lost (failed, or the peer went away): the next plan sends
+ /// to every target that has no record of it, whatever its age.
+ redeliver: bool,
+ /// Advertised to every peer: it is in every mempool that wanted it, so
+ /// nothing is left to send.
+ flooded: bool,
+ /// Peers the body was sent to, and when.
+ sent: Vec<(P, Instant)>,
+ /// Set when a plan found nothing more to send for this transaction: the
+ /// route generation it was planned under, and when the first of those
+ /// sends is old enough to be repeated. Until then, and while the route
+ /// has not changed, a plan skips it without looking at the peers.
+ settled: Option<(u64, Instant)>,
+}
+
+/// Where the transactions go right now.
+#[derive(Debug, PartialEq, Eq)]
+enum Route
{
+ /// Advertise to every peer.
+ Flood,
+ /// Send the body to these peers only (none if this node is the only
+ /// target).
+ Direct(Vec
),
+}
+
+pub struct TxRouter
{
+ /// This node's own validator key, once Core has sent it.
+ local_node: Option,
+ /// Validators that get transactions now; `None` = every peer.
+ targets: Option>,
+ /// Connected peers that proved which validator they are.
+ node_of_peer: HashMap
,
+ /// Transactions this node accepted from its own Core.
+ local: HashMap>,
+ /// The same transactions, oldest first (stale entries are skipped).
+ order: VecDeque<(Instant, TxHash)>,
+ max_local: usize,
+ /// Changes whenever the targets or the identified peers do.
+ generation: u64,
+ /// Ledgers externalized since the router was created (as told by Core).
+ ledgers: u64,
+ /// Transactions peers sent here directly. They were not relayed (the
+ /// sender delivers to every target itself); they are kept so that this
+ /// node can join in, and serve the body, if the sender later has to
+ /// flood one.
+ direct: lru::LruCache,
+ /// Transactions of the last few externalized ledgers.
+ finalized: lru::LruCache,
+}
+
+impl Default for TxRouter
{
+ pub fn new() -> Self {
+ Self::with_capacity(MAX_LOCAL_TXS)
+ }
+
+ pub fn with_capacity(max_local: usize) -> Self {
+ Self {
+ local_node: None,
+ targets: None,
+ node_of_peer: HashMap::new(),
+ local: HashMap::new(),
+ order: VecDeque::new(),
+ max_local,
+ generation: 0,
+ ledgers: 0,
+ direct: lru::LruCache::new(std::num::NonZeroUsize::new(MAX_DIRECT_TXS).unwrap()),
+ finalized: lru::LruCache::new(
+ std::num::NonZeroUsize::new(MAX_FINALIZED_TXS).unwrap(),
+ ),
+ }
+ }
+
+ pub fn set_local_node(&mut self, node: NodeId) {
+ self.local_node = Some(node);
+ }
+
+ /// Replace the targets (`None` = every peer). Returns whether they
+ /// changed; the caller then asks for [`Self::plan_all`].
+ pub fn set_targets(&mut self, targets: Option>) -> bool {
+ if self.targets == targets {
+ return false;
+ }
+ self.targets = targets;
+ self.generation += 1;
+ true
+ }
+
+ /// A connected peer proved that it is `node`.
+ pub fn peer_identified(&mut self, peer: P, node: NodeId) {
+ if self.node_of_peer.insert(peer, node) != Some(node) {
+ self.generation += 1;
+ }
+ }
+
+ pub fn identified_peer_count(&self) -> usize {
+ self.node_of_peer.len()
+ }
+
+ /// The validator a connected peer proved to be.
+ pub fn node_of(&self, peer: &P) -> Option<&NodeId> {
+ self.node_of_peer.get(peer)
+ }
+
+ /// The peer is gone. Whatever was sent to it may not have arrived, and a
+ /// restarted node has an empty mempool: forget what it was sent.
+ pub fn peer_disconnected(&mut self, peer: &P) {
+ self.node_of_peer.remove(peer);
+ self.generation += 1;
+ for entry in self.local.values_mut() {
+ let before = entry.sent.len();
+ entry.sent.retain(|(p, _)| p != peer);
+ if entry.sent.len() != before {
+ entry.redeliver = true;
+ entry.settled = None;
+ }
+ }
+ }
+
+ /// A send to `peer` failed: these transactions were not delivered.
+ pub fn mark_unsent(&mut self, peer: &P, hashes: &[TxHash]) {
+ for hash in hashes {
+ if let Some(entry) = self.local.get_mut(hash) {
+ entry.settled = None;
+ entry.redeliver = true;
+ entry.sent.retain(|(p, _)| p != peer);
+ }
+ }
+ }
+
+ /// Core accepted `tx` from a client. Keeps it and returns what to send
+ /// for it now. A transaction that is already kept is not sent again.
+ pub fn add_local(&mut self, tx: Arc, now: Instant) -> SendPlan
{
+ let hash = *tx.hash();
+ if self.local.contains_key(&hash) {
+ return SendPlan::default();
+ }
+ self.evict_expired(now);
+ while self.local.len() >= self.max_local {
+ if !self.evict_oldest() {
+ break;
+ }
+ }
+ self.order.push_back((now, hash));
+ let mut entry = LocalTx {
+ tx,
+ added: now,
+ added_at_ledger: self.ledgers,
+ planned: false,
+ redeliver: false,
+ flooded: false,
+ sent: Vec::new(),
+ settled: None,
+ };
+ let route = self.route();
+ let mut plan = SendPlan::default();
+ Self::plan_one(
+ &route,
+ self.generation,
+ self.ledgers,
+ &mut entry,
+ now,
+ &mut plan,
+ );
+ self.local.insert(hash, entry);
+ plan
+ }
+
+ /// A peer sent `tx` directly (a body this node did not demand).
+ pub fn note_direct(&mut self, tx: Arc) {
+ self.direct.put(*tx.hash(), DirectTx { tx, relayed: false });
+ }
+
+ /// A peer advertises `hash`. If this node got that transaction directly
+ /// and has not relayed it, its origin has fallen back to flooding:
+ /// returns the transaction to advertise to the other peers, once.
+ pub fn join_flood(&mut self, hash: &TxHash) -> Option> {
+ let entry = self.direct.peek_mut(hash)?;
+ if entry.relayed {
+ return None;
+ }
+ entry.relayed = true;
+ Some(Arc::clone(&entry.tx))
+ }
+
+ /// The body of a transaction this node accepted or received directly,
+ /// for a peer that demands it after it left the short-lived buffer of
+ /// flooded transactions.
+ pub fn body(&self, hash: &TxHash) -> Option> {
+ if let Some(entry) = self.local.get(hash) {
+ return Some(Arc::clone(&entry.tx));
+ }
+ self.direct.peek(hash).map(|entry| Arc::clone(&entry.tx))
+ }
+
+ /// Whether any transaction received directly is being kept.
+ pub fn has_direct(&self) -> bool {
+ !self.direct.is_empty()
+ }
+
+ /// These transactions could not be advertised (no peer was connected):
+ /// they are flooded again at the next plan.
+ pub fn mark_unflooded(&mut self, hashes: &[TxHash]) {
+ for hash in hashes {
+ if let Some(entry) = self.local.get_mut(hash) {
+ entry.settled = None;
+ entry.flooded = false;
+ }
+ }
+ }
+
+ /// Core dropped these transactions (or they are externalized: see
+ /// [`Self::ledger_externalized`]).
+ pub fn remove(&mut self, hashes: &[TxHash]) {
+ for hash in hashes {
+ self.local.remove(hash);
+ self.direct.pop(hash);
+ }
+ // `order` is cleaned lazily: an entry whose transaction is gone, or
+ // was added again later, is skipped when it reaches the front.
+ }
+
+ /// A ledger externalized with these transactions. The caller then asks
+ /// for [`Self::plan_all`]: transactions that were held back from new
+ /// targets may be due now.
+ pub fn ledger_externalized(&mut self, hashes: &[TxHash]) {
+ self.remove(hashes);
+ for hash in hashes {
+ self.finalized.put(*hash, ());
+ }
+ self.ledgers += 1;
+ }
+
+ /// Whether `hash` was in one of the last few externalized ledgers: a
+ /// body that arrives now is a late copy. (This node's mempool does not
+ /// know: it would keep the copy and offer it to the next set it builds,
+ /// where validation against the ledger rejects it.)
+ pub fn was_finalized(&self, hash: &TxHash) -> bool {
+ self.finalized.contains(hash)
+ }
+
+ /// What to send for every kept transaction, after the targets or the
+ /// identified peers changed, or a ledger externalized.
+ pub fn plan_all(&mut self, now: Instant) -> SendPlan
{
+ self.evict_expired(now);
+ let route = self.route();
+ let mut plan = SendPlan::default();
+ // Oldest first, so that a target receives them in arrival order.
+ let hashes: Vec = self
+ .order
+ .iter()
+ .filter(|(added, hash)| self.local.get(hash).is_some_and(|e| e.added == *added))
+ .map(|(_, hash)| *hash)
+ .collect();
+ let generation = self.generation;
+ let ledgers = self.ledgers;
+ for hash in hashes {
+ if let Some(entry) = self.local.get_mut(&hash) {
+ Self::plan_one(&route, generation, ledgers, entry, now, &mut plan);
+ }
+ }
+ plan
+ }
+
+ pub fn len(&self) -> usize {
+ self.local.len()
+ }
+
+ pub fn is_empty(&self) -> bool {
+ self.local.is_empty()
+ }
+
+ pub fn contains(&self, hash: &TxHash) -> bool {
+ self.local.contains_key(hash)
+ }
+
+ /// Whether transactions currently go directly to targets.
+ pub fn is_direct(&self) -> bool {
+ matches!(self.route(), Route::Direct(_))
+ }
+
+ fn route(&self) -> Route
{
+ let Some(targets) = &self.targets else {
+ return Route::Flood;
+ };
+ let mut peers = Vec::new();
+ for target in targets {
+ if self.local_node.as_ref() == Some(target) {
+ continue;
+ }
+ let before = peers.len();
+ for (peer, node) in &self.node_of_peer {
+ if node == target && !peers.contains(peer) {
+ peers.push(*peer);
+ }
+ }
+ if peers.len() == before {
+ // A target this node cannot reach directly: flood, so that
+ // the transaction gets there through other nodes.
+ return Route::Flood;
+ }
+ }
+ Route::Direct(peers)
+ }
+
+ fn plan_one(
+ route: &Route
,
+ ) {
+ if entry.flooded {
+ return;
+ }
+ if let Some((planned, due)) = entry.settled {
+ if planned == generation && now < due {
+ return;
+ }
+ }
+ match route {
+ Route::Flood => {
+ entry.flooded = true;
+ entry.sent.clear();
+ plan.flood.push(Arc::clone(&entry.tx));
+ }
+ Route::Direct(peers) => {
+ // See FOLLOW_TARGETS_MAX_AGE.
+ let follows = !entry.planned
+ || entry.redeliver
+ || now.saturating_duration_since(entry.added) < FOLLOW_TARGETS_MAX_AGE
+ || now.saturating_duration_since(entry.added) >= FOLLOW_TARGETS_ANYWAY_AFTER
+ || ledgers.saturating_sub(entry.added_at_ledger)
+ >= FOLLOW_TARGETS_AFTER_LEDGERS;
+ let mut held_back = false;
+ let mut next_due = now + RESEND_INTERVAL;
+ for peer in peers {
+ let sent_at = entry.sent.iter().find(|(p, _)| p == peer).map(|(_, at)| *at);
+ match sent_at {
+ Some(at) if now.saturating_duration_since(at) < RESEND_INTERVAL => {
+ next_due = next_due.min(at + RESEND_INTERVAL);
+ continue;
+ }
+ // Sent long ago: again (see RESEND_INTERVAL).
+ Some(_) => {}
+ None if !follows => {
+ held_back = true;
+ continue;
+ }
+ None => {}
+ }
+ entry.sent.retain(|(p, _)| p != peer);
+ entry.sent.push((*peer, now));
+ match plan.direct.iter_mut().find(|(p, _)| p == peer) {
+ Some((_, txs)) => txs.push(Arc::clone(&entry.tx)),
+ None => plan.direct.push((*peer, vec![Arc::clone(&entry.tx)])),
+ }
+ }
+ entry.planned = true;
+ entry.redeliver = false;
+ // A transaction that is held back is looked at again by
+ // every plan until it went out or is gone.
+ entry.settled = if held_back {
+ None
+ } else {
+ Some((generation, next_due))
+ };
+ }
+ }
+ }
+
+ fn evict_expired(&mut self, now: Instant) {
+ while let Some((added, hash)) = self.order.front().copied() {
+ let live = self.local.get(&hash).is_some_and(|e| e.added == added);
+ if live && now.saturating_duration_since(added) <= LOCAL_TX_MAX_AGE {
+ break;
+ }
+ self.order.pop_front();
+ if live {
+ self.local.remove(&hash);
+ }
+ }
+ }
+
+ /// Drops the oldest kept transaction. Returns false if there is none.
+ fn evict_oldest(&mut self) -> bool {
+ while let Some((added, hash)) = self.order.pop_front() {
+ if self.local.get(&hash).is_some_and(|e| e.added == added) {
+ self.local.remove(&hash);
+ return true;
+ }
+ }
+ false
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::xdr::tests::valid_transaction_xdr;
+
+ type Router = TxRouter;
+
+ const ME: NodeId = [1; 32];
+ const LEADER1: NodeId = [2; 32];
+ const LEADER2: NodeId = [3; 32];
+ const OTHER: NodeId = [4; 32];
+ const P1: u32 = 10;
+ const P2: u32 = 20;
+ const P3: u32 = 30;
+
+ fn tx(seq: i64) -> Arc {
+ ValidatedTx::from_core_trusted(valid_transaction_xdr(100, seq, 1), 100, 1).unwrap()
+ }
+
+ /// A router that knows itself and three identified peers.
+ fn mesh() -> Router {
+ let mut router = Router::new();
+ router.set_local_node(ME);
+ router.peer_identified(P1, LEADER1);
+ router.peer_identified(P2, LEADER2);
+ router.peer_identified(P3, OTHER);
+ router
+ }
+
+ fn direct_hashes(plan: &SendPlan, peer: u32) -> Vec {
+ plan.direct
+ .iter()
+ .filter(|(p, _)| *p == peer)
+ .flat_map(|(_, txs)| txs.iter().map(|t| *t.hash()))
+ .collect()
+ }
+
+ #[test]
+ fn without_targets_a_transaction_is_flooded_once() {
+ let mut router = mesh();
+ let now = Instant::now();
+ let a = tx(1);
+ let plan = router.add_local(a.clone(), now);
+ assert!(plan.direct.is_empty());
+ assert_eq!(plan.flood.len(), 1);
+ assert_eq!(plan.flood[0].hash(), a.hash());
+ // Flooded: every mempool has it, nothing is ever sent again.
+ assert!(router.plan_all(now).is_empty());
+ router.set_targets(Some(vec![LEADER1, LEADER2]));
+ assert!(router.plan_all(now).is_empty());
+ // Added again: still kept, nothing to send.
+ assert!(router.add_local(a, now).is_empty());
+ assert_eq!(router.len(), 1);
+ }
+
+ #[test]
+ fn with_targets_the_body_goes_to_the_targets_only() {
+ let mut router = mesh();
+ assert!(router.set_targets(Some(vec![LEADER1, LEADER2])));
+ assert!(router.is_direct());
+ let now = Instant::now();
+ let a = tx(1);
+ let plan = router.add_local(a.clone(), now);
+ assert!(plan.flood.is_empty());
+ assert_eq!(direct_hashes(&plan, P1), vec![*a.hash()]);
+ assert_eq!(direct_hashes(&plan, P2), vec![*a.hash()]);
+ assert!(direct_hashes(&plan, P3).is_empty());
+ // Same targets: not sent again.
+ assert!(!router.set_targets(Some(vec![LEADER1, LEADER2])));
+ assert!(router.plan_all(now).is_empty());
+ }
+
+ #[test]
+ fn a_pending_transaction_follows_the_targets() {
+ let mut router = mesh();
+ router.set_targets(Some(vec![LEADER1, LEADER2]));
+ let now = Instant::now();
+ let a = tx(1);
+ let b = tx(2);
+ router.add_local(a.clone(), now);
+ router.add_local(b.clone(), now);
+ // `a` is externalized; `b` was not included, and not in the two
+ // ledgers after it either.
+ router.ledger_externalized(&[*a.hash()]);
+ assert!(router.was_finalized(a.hash()));
+ assert!(!router.was_finalized(b.hash()));
+ router.ledger_externalized(&[]);
+ router.ledger_externalized(&[]);
+ assert!(router.set_targets(Some(vec![OTHER, LEADER1])));
+ let plan = router.plan_all(now + Duration::from_secs(1));
+ // The new target gets `b`; the one that stays a target has it.
+ assert_eq!(direct_hashes(&plan, P3), vec![*b.hash()]);
+ assert!(direct_hashes(&plan, P1).is_empty());
+ assert!(direct_hashes(&plan, P2).is_empty());
+ assert!(plan.flood.is_empty());
+ // Back to an earlier target long after: it is sent again.
+ router.set_targets(Some(vec![LEADER2]));
+ let later = now + RESEND_INTERVAL + Duration::from_secs(1);
+ let plan = router.plan_all(later);
+ assert_eq!(direct_hashes(&plan, P2), vec![*b.hash()]);
+ }
+
+ #[test]
+ fn this_node_as_a_target_needs_no_send() {
+ let mut router = mesh();
+ router.set_targets(Some(vec![ME, LEADER2]));
+ let now = Instant::now();
+ let plan = router.add_local(tx(1), now);
+ assert_eq!(plan.direct.len(), 1);
+ assert_eq!(plan.direct[0].0, P2);
+ // The only target: kept, nothing to send, and not flooded.
+ router.set_targets(Some(vec![ME]));
+ assert!(router.is_direct());
+ let plan = router.add_local(tx(2), now);
+ assert!(plan.is_empty());
+ assert_eq!(router.len(), 2);
+ }
+
+ #[test]
+ fn an_unreachable_target_means_flooding() {
+ let mut router = Router::new();
+ router.set_local_node(ME);
+ router.peer_identified(P1, LEADER1);
+ // LEADER2 is not a connected, identified peer.
+ router.set_targets(Some(vec![LEADER1, LEADER2]));
+ assert!(!router.is_direct());
+ let now = Instant::now();
+ let a = tx(1);
+ let plan = router.add_local(a.clone(), now);
+ assert!(plan.direct.is_empty());
+ assert_eq!(plan.flood.len(), 1);
+ }
+
+ #[test]
+ fn a_transaction_sent_directly_is_flooded_when_a_target_becomes_unreachable() {
+ let mut router = mesh();
+ router.set_targets(Some(vec![LEADER1, LEADER2]));
+ let now = Instant::now();
+ let a = tx(1);
+ assert_eq!(router.add_local(a.clone(), now).direct.len(), 2);
+ // The next targets include a node that never identified itself.
+ router.set_targets(Some(vec![[9; 32], LEADER1]));
+ let plan = router.plan_all(now);
+ assert!(plan.direct.is_empty());
+ assert_eq!(plan.flood.len(), 1);
+ assert_eq!(plan.flood[0].hash(), a.hash());
+ // Once flooded it is done.
+ router.set_targets(Some(vec![LEADER1, LEADER2]));
+ assert!(router.plan_all(now).is_empty());
+ }
+
+ #[test]
+ fn identification_completes_the_route_and_sends_what_is_pending() {
+ let mut router = Router::new();
+ router.set_local_node(ME);
+ router.set_targets(Some(vec![LEADER1]));
+ assert!(!router.is_direct());
+ router.peer_identified(P1, LEADER1);
+ assert!(router.is_direct());
+ let now = Instant::now();
+ let a = tx(1);
+ assert_eq!(direct_hashes(&router.add_local(a.clone(), now), P1), vec![*a.hash()]);
+ }
+
+ #[test]
+ fn a_transaction_follows_new_targets_only_while_fresh_or_once_it_missed_three_ledgers() {
+ let mut router = mesh();
+ router.set_targets(Some(vec![LEADER1, LEADER2]));
+ let start = Instant::now();
+ let old = tx(1);
+ assert_eq!(router.add_local(old.clone(), start).direct.len(), 2);
+
+ // The targets change while `old` is most likely in the set its
+ // targets are voting on; `fresh` was accepted just before the change
+ // and may have missed that set.
+ let change = start + Duration::from_millis(700);
+ let fresh = tx(2);
+ router.add_local(fresh.clone(), change - Duration::from_millis(100));
+ let stale = tx(3);
+ router.add_local(
+ stale.clone(),
+ change - FOLLOW_TARGETS_MAX_AGE - Duration::from_millis(1),
+ );
+ assert!(router.set_targets(Some(vec![LEADER1, OTHER])));
+ let plan = router.plan_all(change);
+ assert_eq!(direct_hashes(&plan, P3), vec![*fresh.hash()]);
+ assert!(direct_hashes(&plan, P1).is_empty());
+ // Asking again changes nothing: `old` is still held back.
+ assert!(router.plan_all(change + Duration::from_millis(50)).is_empty());
+
+ // The ledger that was in consensus externalizes without `old`: its
+ // set may have been built before `old` was accepted. The next one
+ // may have been built before `old` reached the leader.
+ router.ledger_externalized(&[]);
+ assert!(router.plan_all(change + Duration::from_millis(400)).is_empty());
+ router.ledger_externalized(&[]);
+ assert!(router.plan_all(change + Duration::from_millis(1200)).is_empty());
+ // The third was built with `old` in the mempool and did not have it
+ // either: now it goes to the new target, once, and so does the one
+ // that was just too old at the change.
+ router.ledger_externalized(&[]);
+ let plan = router.plan_all(change + Duration::from_millis(2000));
+ assert_eq!(direct_hashes(&plan, P3), vec![*old.hash(), *stale.hash()]);
+ assert!(direct_hashes(&plan, P1).is_empty());
+ assert!(router.plan_all(change + Duration::from_millis(2100)).is_empty());
+ }
+
+ #[test]
+ fn a_held_back_transaction_follows_the_targets_when_no_ledger_closes() {
+ let mut router = mesh();
+ router.set_targets(Some(vec![LEADER1, LEADER2]));
+ let start = Instant::now();
+ let a = tx(1);
+ router.add_local(a.clone(), start);
+ // Two rounds time out: the leader of the third round is new.
+ assert!(router.set_targets(Some(vec![LEADER2, OTHER])));
+ assert!(router.plan_all(start + Duration::from_secs(2)).is_empty());
+ let plan = router.plan_all(start + FOLLOW_TARGETS_ANYWAY_AFTER);
+ assert_eq!(direct_hashes(&plan, P3), vec![*a.hash()]);
+ assert!(direct_hashes(&plan, P2).is_empty());
+ }
+
+ #[test]
+ fn a_removal_by_core_is_not_a_ledger() {
+ let mut router = mesh();
+ router.set_targets(Some(vec![LEADER1]));
+ let start = Instant::now();
+ let a = tx(1);
+ let dropped = tx(2);
+ router.add_local(a.clone(), start);
+ router.add_local(dropped.clone(), start);
+ router.set_targets(Some(vec![LEADER1, LEADER2]));
+ // Core drops transactions for its own reasons, any number of times:
+ // that neither counts as a ledger nor marks them final.
+ for _ in 0..5 {
+ router.remove(&[*dropped.hash()]);
+ }
+ assert!(!router.was_finalized(dropped.hash()));
+ assert!(router.plan_all(start + Duration::from_secs(1)).is_empty());
+ }
+
+ #[test]
+ fn a_transaction_that_externalizes_is_never_sent_to_the_new_targets() {
+ let mut router = mesh();
+ router.set_targets(Some(vec![LEADER1]));
+ let start = Instant::now();
+ let a = tx(1);
+ router.add_local(a.clone(), start);
+ // The node closes its ledger late: the targets change 600 ms after
+ // `a` went to the leader, which has it in the set in consensus.
+ assert!(router.set_targets(Some(vec![LEADER1, LEADER2])));
+ assert!(router.plan_all(start + Duration::from_millis(600)).is_empty());
+ // That set externalizes two ledgers later (this node sees ledgers
+ // late): `a` is gone, and LEADER2 never got a copy that could have
+ // arrived after it.
+ router.ledger_externalized(&[]);
+ assert!(router.plan_all(start + Duration::from_millis(900)).is_empty());
+ router.ledger_externalized(&[]);
+ assert!(router.plan_all(start + Duration::from_millis(1700)).is_empty());
+ router.ledger_externalized(&[*a.hash()]);
+ assert!(router.plan_all(start + Duration::from_secs(2)).is_empty());
+ assert!(router.was_finalized(a.hash()));
+ }
+
+ #[test]
+ fn a_transaction_this_node_led_follows_the_targets_under_the_same_rule() {
+ let mut router = mesh();
+ // This node is the only target: nothing to send.
+ router.set_targets(Some(vec![ME]));
+ let start = Instant::now();
+ let a = tx(1);
+ assert!(router.add_local(a.clone(), start).is_empty());
+ // Its term ends while `a` may be in its own set in consensus.
+ router.set_targets(Some(vec![LEADER1]));
+ assert!(router.plan_all(start + Duration::from_millis(600)).is_empty());
+ router.ledger_externalized(&[]);
+ router.ledger_externalized(&[]);
+ router.ledger_externalized(&[]);
+ let plan = router.plan_all(start + Duration::from_secs(2));
+ assert_eq!(direct_hashes(&plan, P1), vec![*a.hash()]);
+ }
+
+ #[test]
+ fn a_reconnected_target_gets_the_pending_transactions_again() {
+ let mut router = mesh();
+ router.set_targets(Some(vec![LEADER1]));
+ let now = Instant::now();
+ let a = tx(1);
+ assert_eq!(router.add_local(a.clone(), now).direct.len(), 1);
+ router.set_targets(Some(vec![LEADER2]));
+ router.plan_all(now);
+ // LEADER1 restarts while it is not a target, then is one again,
+ // within the interval in which nothing is sent twice.
+ router.peer_disconnected(&P1);
+ assert!(router.node_of(&P1).is_none());
+ router.peer_identified(P1, LEADER1);
+ router.set_targets(Some(vec![LEADER1]));
+ let plan = router.plan_all(now + Duration::from_secs(1));
+ assert_eq!(direct_hashes(&plan, P1), vec![*a.hash()]);
+ }
+
+ #[test]
+ fn a_failed_send_is_retried_at_the_next_plan() {
+ let mut router = mesh();
+ router.set_targets(Some(vec![LEADER1]));
+ let now = Instant::now();
+ let a = tx(1);
+ router.add_local(a.clone(), now);
+ assert!(router.plan_all(now).is_empty());
+ router.mark_unsent(&P1, &[*a.hash()]);
+ assert_eq!(direct_hashes(&router.plan_all(now), P1), vec![*a.hash()]);
+ }
+
+ #[test]
+ fn two_peers_of_one_validator_both_get_the_body() {
+ let mut router = mesh();
+ router.peer_identified(P3, LEADER1);
+ router.set_targets(Some(vec![LEADER1]));
+ let plan = router.add_local(tx(1), Instant::now());
+ let mut peers: Vec = plan.direct.iter().map(|(p, _)| *p).collect();
+ peers.sort();
+ assert_eq!(peers, vec![P1, P3]);
+ }
+
+ #[test]
+ fn kept_transactions_are_bounded_and_expire() {
+ let mut router = Router::with_capacity(2);
+ router.set_local_node(ME);
+ router.peer_identified(P1, LEADER1);
+ router.set_targets(Some(vec![OTHER])); // unreachable: no sends recorded
+ let now = Instant::now();
+ let (a, b, c) = (tx(1), tx(2), tx(3));
+ router.add_local(a.clone(), now);
+ router.add_local(b.clone(), now + Duration::from_secs(1));
+ router.add_local(c.clone(), now + Duration::from_secs(2));
+ // The oldest was dropped for the third.
+ assert_eq!(router.len(), 2);
+ assert!(!router.contains(a.hash()));
+ assert!(router.contains(b.hash()) && router.contains(c.hash()));
+ // Age: `b` expires first.
+ router.plan_all(now + Duration::from_secs(1) + LOCAL_TX_MAX_AGE + Duration::from_millis(1));
+ assert!(!router.contains(b.hash()));
+ assert!(router.contains(c.hash()));
+ router.plan_all(now + Duration::from_secs(2) + LOCAL_TX_MAX_AGE + Duration::from_millis(1));
+ assert!(router.is_empty());
+ }
+
+ #[test]
+ fn a_directly_received_transaction_joins_a_flood_once_and_can_be_served() {
+ let mut router = mesh();
+ let (a, b) = (tx(1), tx(2));
+ assert!(!router.has_direct());
+ router.note_direct(a.clone());
+ assert!(router.has_direct());
+ // Not advertised by anyone: nothing to do. Kept, so a demand for it
+ // can be answered.
+ assert!(router.join_flood(b.hash()).is_none());
+ assert_eq!(router.body(a.hash()).unwrap().hash(), a.hash());
+ // First advert: join the flood. Later adverts: already done.
+ assert_eq!(router.join_flood(a.hash()).unwrap().hash(), a.hash());
+ assert!(router.join_flood(a.hash()).is_none());
+ assert!(router.body(a.hash()).is_some());
+ // Externalized: forgotten.
+ router.remove(&[*a.hash()]);
+ assert!(router.body(a.hash()).is_none());
+ assert!(!router.has_direct());
+ // A transaction this node accepted itself can be served as well.
+ router.add_local(b.clone(), Instant::now());
+ assert!(router.body(b.hash()).is_some());
+ }
+
+ #[test]
+ fn a_plan_with_unchanged_targets_repeats_a_send_only_after_the_interval() {
+ // With every validator as a target the list never changes: a body a
+ // peer dropped is only recovered by the periodic plan.
+ let mut router = mesh();
+ router.set_targets(Some(vec![LEADER1, LEADER2, OTHER]));
+ let now = Instant::now();
+ let a = tx(1);
+ assert_eq!(router.add_local(a.clone(), now).direct.len(), 3);
+ for secs in [1, 10, 59] {
+ assert!(router.plan_all(now + Duration::from_secs(secs)).is_empty());
+ }
+ let plan = router.plan_all(now + RESEND_INTERVAL);
+ assert_eq!(plan.direct.len(), 3);
+ assert!(router.plan_all(now + RESEND_INTERVAL + Duration::from_secs(1)).is_empty());
+ // A failed send is repeated at the next plan, whatever the interval.
+ router.mark_unsent(&P2, &[*a.hash()]);
+ let plan = router.plan_all(now + RESEND_INTERVAL + Duration::from_secs(2));
+ assert_eq!(direct_hashes(&plan, P2), vec![*a.hash()]);
+ assert!(direct_hashes(&plan, P1).is_empty());
+ }
+
+ #[test]
+ fn a_flood_that_reached_nobody_is_repeated() {
+ let mut router = Router::new();
+ let now = Instant::now();
+ let a = tx(1);
+ assert_eq!(router.add_local(a.clone(), now).flood.len(), 1);
+ assert!(router.plan_all(now).is_empty());
+ // No peer was connected when it was to be advertised.
+ router.mark_unflooded(&[*a.hash()]);
+ assert_eq!(router.plan_all(now).flood.len(), 1);
+ assert!(router.plan_all(now).is_empty());
+ }
+
+ #[test]
+ fn a_removed_transaction_added_again_is_a_new_one() {
+ let mut router = mesh();
+ router.set_targets(Some(vec![LEADER1]));
+ let now = Instant::now();
+ let a = tx(1);
+ router.add_local(a.clone(), now);
+ router.remove(&[*a.hash()]);
+ assert!(router.is_empty());
+ assert!(router.plan_all(now).is_empty());
+ // Submitted again later: sent again, and it expires by its new age.
+ let later = now + Duration::from_secs(200);
+ assert_eq!(router.add_local(a.clone(), later).direct.len(), 1);
+ router.plan_all(now + LOCAL_TX_MAX_AGE + Duration::from_secs(1));
+ assert!(router.contains(a.hash()));
+ router.plan_all(later + LOCAL_TX_MAX_AGE + Duration::from_secs(1));
+ assert!(router.is_empty());
+ }
+}
diff --git a/overlay/src/ipc/messages.rs b/overlay/src/ipc/messages.rs
index d326a82e3..7ffb64682 100644
--- a/overlay/src/ipc/messages.rs
+++ b/overlay/src/ipc/messages.rs
@@ -62,6 +62,17 @@ pub enum MessageType {
/// Payload: [hash:32][slotSeq:4]
PushTxSet = 14,
+ /// This node's validator identity, in answer to `OverlayIdentity`: the
+ /// node key's signature over
+ /// `SHA256(networkID || "stellar-overlay-peer-id" || peerId)`.
+ /// Payload: [networkID:32][nodeID:32][signature:64]
+ SetNodeIdentity = 15,
+
+ /// The validators that get this node's transactions now.
+ /// Payload: [everyone:1][count:4][nodeID:32]...
+ /// `everyone` = 1: flood to all peers (the list is ignored).
+ SetTxTargets = 16,
+
// ═══ Overlay → Core (Critical Path) ═══
/// Received SCP envelope from network
ScpReceived = 100,
@@ -80,6 +91,11 @@ pub enum MessageType {
/// Overlay metrics snapshot response (JSON payload)
OverlayMetricsResponse = 105,
+
+ /// The overlay's libp2p peer id, sent once at startup so that Core can
+ /// sign the identity binding (`SetNodeIdentity`).
+ /// Payload: [peerId bytes]
+ OverlayIdentity = 106,
}
impl TryFrom for MessageType {
@@ -100,11 +116,14 @@ impl TryFrom for MessageType {
12 => Ok(MessageType::CacheTxSet),
13 => Ok(MessageType::RequestOverlayMetrics),
14 => Ok(MessageType::PushTxSet),
+ 15 => Ok(MessageType::SetNodeIdentity),
+ 16 => Ok(MessageType::SetTxTargets),
100 => Ok(MessageType::ScpReceived),
101 => Ok(MessageType::TopTxsResponse),
102 => Ok(MessageType::PeerRequestsScpState),
103 => Ok(MessageType::TxSetAvailable),
105 => Ok(MessageType::OverlayMetricsResponse),
+ 106 => Ok(MessageType::OverlayIdentity),
_ => Err(InvalidMessageType(value)),
}
}
@@ -302,6 +321,9 @@ mod tests {
MessageType::CacheTxSet,
MessageType::RequestOverlayMetrics,
MessageType::PushTxSet,
+ MessageType::SetNodeIdentity,
+ MessageType::SetTxTargets,
+ MessageType::OverlayIdentity,
MessageType::ScpReceived,
MessageType::TopTxsResponse,
MessageType::PeerRequestsScpState,
@@ -373,6 +395,15 @@ mod tests {
MessageType::RequestOverlayMetrics
);
assert_eq!(MessageType::try_from(14).unwrap(), MessageType::PushTxSet);
+ assert_eq!(
+ MessageType::try_from(15).unwrap(),
+ MessageType::SetNodeIdentity
+ );
+ assert_eq!(MessageType::try_from(16).unwrap(), MessageType::SetTxTargets);
+ assert_eq!(
+ MessageType::try_from(106).unwrap(),
+ MessageType::OverlayIdentity
+ );
assert_eq!(
MessageType::try_from(100).unwrap(),
MessageType::ScpReceived
@@ -401,7 +432,8 @@ mod tests {
assert!(MessageType::try_from(9).is_err()); // gap between 8 and 10
assert!(MessageType::try_from(99).is_err());
assert!(MessageType::try_from(104).is_err());
- assert!(MessageType::try_from(106).is_err());
+ assert!(MessageType::try_from(17).is_err());
+ assert!(MessageType::try_from(107).is_err());
assert!(MessageType::try_from(u32::MAX).is_err());
}
}
diff --git a/overlay/src/libp2p_overlay.rs b/overlay/src/libp2p_overlay.rs
index d936fd513..effb1017e 100644
--- a/overlay/src/libp2p_overlay.rs
+++ b/overlay/src/libp2p_overlay.rs
@@ -12,8 +12,8 @@
//! QUIC provides independent loss recovery per stream.
use crate::flood::{
- GetData, InvBatch, InvBatcher, InvEntry, InvTracker, PendingRequests, PushRateLimiter,
- PushRecord, TxBuffer, TxSetData, TxStreamMessage,
+ GetData, InvBatch, InvBatcher, InvEntry, InvTracker, NodeId, PendingRequests, PushRateLimiter,
+ PushRecord, SendPlan, TxBuffer, TxRouter, TxSetData, TxStreamMessage,
};
#[cfg(test)]
use crate::integrated::CoreCommand;
@@ -47,6 +47,100 @@ pub const CONTROL_PROTOCOL: StreamProtocol = StreamProtocol::new("/stellar/contr
pub const TX_PROTOCOL: StreamProtocol = StreamProtocol::new("/stellar/tx/1.0.0");
// This experimental protocol requires zstd for every transaction-set response.
pub const TXSET_PROTOCOL: StreamProtocol = StreamProtocol::new("/stellar/txset/zstd/2.0.0");
+/// One-shot stream on which an overlay tells a peer which validator it
+/// belongs to: one frame, `[nodeID:32][signature:64]` (see `NodeIdentity`),
+/// answered with one byte once it has been read.
+pub const NODE_ID_PROTOCOL: StreamProtocol = StreamProtocol::new("/stellar/node-id/1.0.0");
+
+/// Domain separator of the identity binding that Core signs.
+const NODE_ID_BINDING_TAG: &[u8] = b"stellar-overlay-peer-id";
+const NODE_ID_FRAME_LEN: usize = 32 + 64;
+
+/// Transaction bodies sent directly to a peer are collected for this long,
+/// so that a burst of submissions (a client or load generator step) becomes
+/// one write per peer instead of one per transaction.
+const DIRECT_SEND_BATCH_INTERVAL: Duration = Duration::from_millis(10);
+
+/// A peer's queue of direct sends is written at once when it reaches this
+/// many transactions.
+const DIRECT_SEND_BATCH_MAX_TXS: usize = 256;
+
+/// At most this many transactions wait for one peer whose stream is not
+/// taking them (one sender per peer writes at a time). What does not fit is
+/// handed back to the router as not sent.
+const DIRECT_SEND_MAX_QUEUED_TXS: usize = 20_000;
+
+/// How often the router is asked whether anything is owed although nothing
+/// changed: a send that failed, a body old enough to be sent again, a flood
+/// that found no peer.
+const DIRECT_REPLAN_INTERVAL: Duration = Duration::from_secs(2);
+
+/// Direct sends per peer: what is queued, and which peers have a sender task
+/// writing right now.
+#[derive(Default)]
+struct DirectOut {
+ queued: HashMap>>,
+ sending: HashSet,
+}
+
+/// A peer's validator identity claim must arrive within this long of its
+/// stream opening.
+const NODE_ID_READ_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// Acknowledgement bytes on the node id stream: the claim was dealt with
+/// (accepted, kept for later, or refused), or the receiver could not use it
+/// yet and the sender should repeat it.
+const NODE_ID_ACK_TAKEN: u8 = 1;
+const NODE_ID_ACK_RETRY: u8 = 0;
+
+/// Identity claims handled per connection. An honest peer sends one per
+/// connection it establishes (two when both sides dial at once) and one more
+/// if its Core's identity arrives late.
+const MAX_NODE_ID_CLAIMS_PER_PEER: u32 = 8;
+
+/// This node's validator identity, as sent by Core: the node key's signature
+/// over [`node_id_binding`] of this overlay's own peer id.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct NodeIdentity {
+ pub network_id: [u8; 32],
+ pub node_id: NodeId,
+ pub signature: [u8; 64],
+}
+
+/// What a validator signs to say "the overlay with this peer id is mine":
+/// `SHA256(networkID || tag || peerId)`. The peer id is the one QUIC
+/// authenticated, so the statement cannot be replayed by another peer.
+pub fn node_id_binding(network_id: &[u8; 32], peer: &PeerId) -> [u8; 32] {
+ let peer_bytes = peer.to_bytes();
+ let mut message = Vec::with_capacity(32 + NODE_ID_BINDING_TAG.len() + peer_bytes.len());
+ message.extend_from_slice(network_id);
+ message.extend_from_slice(NODE_ID_BINDING_TAG);
+ message.extend_from_slice(&peer_bytes);
+ crate::xdr::sha256_hash(&message)
+}
+
+/// Whether `signature` is `node_id`'s signature binding it to `peer`.
+fn verify_node_id_binding(
+ network_id: &[u8; 32],
+ peer: &PeerId,
+ node_id: &NodeId,
+ signature: &[u8; 64],
+) -> bool {
+ let Ok(key) = libp2p::identity::ed25519::PublicKey::try_from_bytes(node_id) else {
+ return false;
+ };
+ key.verify(&node_id_binding(network_id, peer), signature)
+}
+
+/// Identity state: ours, and claims from peers that arrived before Core told
+/// us the network id needed to check them.
+#[derive(Default)]
+struct IdentityState {
+ ours: Option,
+ unverified: HashMap,
+ /// Claims handled per connected peer (see MAX_NODE_ID_CLAIMS_PER_PEER).
+ claims: HashMap,
+}
/// Message frame: 4-byte length prefix + payload
/// Max message size: 16MB (for large TX sets)
@@ -120,6 +214,14 @@ pub enum OverlayCommand {
hash: [u8; 32],
data: Arc,
},
+ /// Core accepted this transaction. The router already holds it; this
+ /// carries out what it decided at that moment.
+ SendLocalTx(SendPlan),
+ /// The targets or the identified peers changed: send what the router now
+ /// says is owed.
+ FlushLocalTxs,
+ /// Core sent this node's validator identity: tell every peer.
+ SetNodeIdentity(NodeIdentity),
/// Connect to a peer by address (bootstrap — PeerId unknown)
Dial(Multiaddr),
/// Connect to a known peer by PeerId (reconnect — deduplicates automatically)
@@ -196,6 +298,11 @@ pub struct OverlayHandle {
cmd_tx: mpsc::Sender,
txset_send_slots: Arc,
txset_send_bytes: Arc,
+ /// Shared with the overlay task. The App updates it in the order of
+ /// Core's messages (a transaction accepted, transactions externalized,
+ /// new targets), then sends a command to act on it.
+ tx_router: Arc>>,
+ local_peer_id: PeerId,
}
impl OverlayHandle {
@@ -221,6 +328,84 @@ impl OverlayHandle {
}
}
+ /// This overlay's own libp2p peer id (what Core's identity binding signs).
+ pub fn local_peer_id(&self) -> PeerId {
+ self.local_peer_id
+ }
+
+ /// Core accepted `tx`: keep it until it is externalized, and work out
+ /// where it goes now. Synchronous, so that it is ordered with
+ /// [`Self::ledger_externalized`] and [`Self::set_tx_targets`]; the returned
+ /// plan is carried out with [`Self::send_local_tx`].
+ pub fn add_local_tx(&self, tx: Arc) -> SendPlan {
+ self.tx_router
+ .lock()
+ .unwrap()
+ .add_local(tx, Instant::now())
+ }
+
+ /// Carry out a plan from [`Self::add_local_tx`].
+ pub async fn send_local_tx(&self, plan: SendPlan) {
+ if plan.is_empty() {
+ return;
+ }
+ if let Err(e) = self.cmd_tx.send(OverlayCommand::SendLocalTx(plan)).await {
+ warn!(
+ "Overlay command channel closed, failed to send SendLocalTx: {}",
+ e
+ );
+ }
+ }
+
+ /// A ledger externalized with these transactions: stop sending them.
+ /// The caller then calls [`Self::flush_local_txs`]: transactions that
+ /// were held back from new targets may be due.
+ pub fn ledger_externalized(&self, hashes: &[[u8; 32]]) {
+ self.tx_router.lock().unwrap().ledger_externalized(hashes);
+ }
+
+ /// Core dropped these transactions: stop sending them.
+ pub fn forget_local_txs(&self, hashes: &[[u8; 32]]) {
+ self.tx_router.lock().unwrap().remove(hashes);
+ }
+
+ /// Core's list of validators that get transactions now (`None` = every
+ /// peer). Returns whether it changed; if so, call
+ /// [`Self::flush_local_txs`].
+ pub fn set_tx_targets(&self, targets: Option>) -> bool {
+ self.tx_router.lock().unwrap().set_targets(targets)
+ }
+
+ /// Send whatever the router says is owed after a change.
+ pub async fn flush_local_txs(&self) {
+ if let Err(e) = self.cmd_tx.send(OverlayCommand::FlushLocalTxs).await {
+ warn!(
+ "Overlay command channel closed, failed to send FlushLocalTxs: {}",
+ e
+ );
+ }
+ }
+
+ /// Core sent this node's validator identity.
+ pub async fn set_node_identity(&self, identity: NodeIdentity) {
+ if let Err(e) = self
+ .cmd_tx
+ .send(OverlayCommand::SetNodeIdentity(identity))
+ .await
+ {
+ warn!(
+ "Overlay command channel closed, failed to send SetNodeIdentity: {}",
+ e
+ );
+ }
+ }
+
+ /// Number of transactions kept for sending (accepted from Core, not yet
+ /// externalized).
+ pub fn local_tx_count(&self) -> usize {
+ self.tx_router.lock().unwrap().len()
+ }
+
pub async fn fetch_txset(&self, hash: [u8; 32], slot: u32) {
self.send_fetch_txset(hash, slot, false).await;
}
@@ -447,6 +632,15 @@ struct SharedState {
pending_getdata: RwLock,
/// TX buffer for responding to GETDATA requests
tx_buffer: RwLock,
+ /// Where this node's own transactions go (shared with the App through
+ /// `OverlayHandle`). Never held across an await.
+ tx_router: Arc>>,
+ /// Validator identities: ours and unchecked claims. Never held across
+ /// an await.
+ identity: std::sync::Mutex,
+ /// Transaction bodies queued for a direct send, per peer (see
+ /// DIRECT_SEND_BATCH_INTERVAL). Never held across an await.
+ direct_out: std::sync::Mutex,
/// Overlay metrics (shared with App for IPC reporting)
metrics: Arc,
}
@@ -457,6 +651,7 @@ impl SharedState {
mempool: MempoolHandle,
control: Control,
metrics: Arc,
+ tx_router: Arc>>,
) -> Self {
Self {
peer_streams: RwLock::new(HashMap::new()),
@@ -484,6 +679,9 @@ impl SharedState {
inv_tracker: RwLock::new(InvTracker::new()),
pending_getdata: RwLock::new(PendingRequests::new()),
tx_buffer: RwLock::new(TxBuffer::new()),
+ tx_router,
+ identity: std::sync::Mutex::new(IdentityState::default()),
+ direct_out: std::sync::Mutex::new(DirectOut::default()),
metrics,
}
}
@@ -587,11 +785,13 @@ pub fn create_overlay(
// Unbounded channel for critical events (SCP, TxSet) - never drop
let (event_tx, event_rx) = mpsc::unbounded_channel();
+ let tx_router = Arc::new(std::sync::Mutex::new(TxRouter::new()));
let state = Arc::new(SharedState::new(
event_tx,
mempool,
control.clone(),
metrics,
+ Arc::clone(&tx_router),
));
let overlay = StellarOverlay {
@@ -606,6 +806,8 @@ pub fn create_overlay(
cmd_tx,
txset_send_slots: Arc::new(Semaphore::new(MAX_OUTSTANDING_TXSET_SENDS)),
txset_send_bytes: Arc::new(Semaphore::new(MAX_OUTSTANDING_TXSET_BYTES)),
+ tx_router,
+ local_peer_id: peer_id,
};
Ok((handle, event_rx, overlay))
@@ -646,13 +848,22 @@ impl StellarOverlay {
.control
.accept(TXSET_PROTOCOL)
.expect("TxSet protocol registered once");
+ let node_id_incoming = self
+ .control
+ .accept(NODE_ID_PROTOCOL)
+ .expect("node id protocol registered once");
let state = self.state.clone();
+ tokio::spawn(handle_inbound_node_id_streams(
+ node_id_incoming,
+ state.clone(),
+ ));
tokio::spawn(handle_inbound_scp_streams(control_incoming, state.clone()));
tokio::spawn(handle_inbound_tx_streams(tx_incoming, state.clone()));
tokio::spawn(handle_inbound_txset_streams(txset_incoming, state.clone()));
// Spawn INV/GETDATA housekeeping task
tokio::spawn(inv_getdata_housekeeping_task(state.clone()));
+ tokio::spawn(direct_send_flush_task(state.clone()));
loop {
tokio::select! {
@@ -694,6 +905,15 @@ impl StellarOverlay {
OverlayCommand::PushTxSet { hash, data } => {
self.push_txset(hash, data).await;
}
+ OverlayCommand::SendLocalTx(plan) => {
+ self.send_local_tx_plan(plan).await;
+ }
+ OverlayCommand::FlushLocalTxs => {
+ flush_local_txs(&self.state).await;
+ }
+ OverlayCommand::SetNodeIdentity(identity) => {
+ self.set_node_identity(identity).await;
+ }
OverlayCommand::Dial(addr) => {
info!("Dialing peer at {}", addr);
self.state.metrics.connection_pending.fetch_add(1, Ordering::Relaxed);
@@ -827,6 +1047,16 @@ impl StellarOverlay {
"Duplicate connection to {} (now {}), skipping stream setup",
peer_id, num_established
);
+ // The peer may have seen its side of the earlier
+ // connection close and forgotten who this node is.
+ let identity = self.state.identity.lock().unwrap().ours.clone();
+ if let Some(identity) = identity {
+ tokio::spawn(send_node_identity(
+ self.control.clone(),
+ peer_id,
+ identity,
+ ));
+ }
}
}
@@ -877,6 +1107,26 @@ impl StellarOverlay {
.lock()
.unwrap()
.forget_peer(&peer_id);
+ // The peer is no longer a reachable target, and whatever
+ // was sent to it may be lost.
+ {
+ let mut ids = self.state.identity.lock().unwrap();
+ ids.unverified.remove(&peer_id);
+ ids.claims.remove(&peer_id);
+ }
+ // Whatever was queued for the peer is not sent; the
+ // router forgets its sends to the peer just below.
+ self.state.direct_out.lock().unwrap().queued.remove(&peer_id);
+ let identified = {
+ let mut router = self.state.tx_router.lock().unwrap();
+ router.peer_disconnected(&peer_id);
+ router.identified_peer_count()
+ };
+ self.state
+ .metrics
+ .node_id_identified
+ .store(identified as i64, Ordering::Relaxed);
+ flush_local_txs(&self.state).await;
// Notify main loop to clean up any pending requests for this peer
if let Err(e) = self.state.event_tx.send(OverlayEvent::PeerDisconnected {
peer_id: peer_id.clone(),
@@ -1014,11 +1264,11 @@ impl StellarOverlay {
}
}
- /// Broadcast TX to all connected peers
- /// Broadcast TX using INV/GETDATA protocol (bandwidth efficient)
+ /// Flood a transaction to all connected peers with the INV/GETDATA
+ /// protocol, unless it was seen before. (Core's own transactions go
+ /// through the router instead, see `send_local_tx_plan`.)
async fn broadcast_tx(&mut self, tx: Arc) {
let hash = *tx.hash();
- let fee_per_op = tx.fee_per_op();
// Dedup check
{
@@ -1033,34 +1283,57 @@ impl StellarOverlay {
.memory_flood_known
.store(seen.len() as i64, Ordering::Relaxed);
}
+ advertise_txs_to_all(&self.state, &[tx]).await;
+ }
- // Store TX in buffer for GETDATA responses
+ /// Carry out the router's plan for transactions Core accepted.
+ async fn send_local_tx_plan(&mut self, plan: SendPlan) {
{
- let mut buffer = self.state.tx_buffer.write().await;
- buffer.insert(Arc::clone(&tx));
- }
-
- let streams = self.state.peer_streams.read().await;
- let peers: Vec<_> = streams.keys().cloned().collect();
- drop(streams);
-
- if peers.is_empty() {
- debug!("TX_INV: No peers to announce TX {:02x?}...", &hash[..4]);
- return;
+ // A local transaction is known from now on: a copy from a peer
+ // is a duplicate.
+ let mut seen = self.state.tx_seen.write().await;
+ for tx in plan
+ .flood
+ .iter()
+ .chain(plan.direct.iter().flat_map(|(_, txs)| txs.iter()))
+ {
+ seen.put(*tx.hash(), ());
+ }
+ self.state
+ .metrics
+ .memory_flood_known
+ .store(seen.len() as i64, Ordering::Relaxed);
}
+ execute_send_plan(&self.state, plan).await;
+ }
- debug!(
- "TX_INV: Announcing TX {:02x?}... ({} bytes) to {} peers via INV",
- &hash[..4],
- tx.bytes().len(),
- peers.len()
+ /// Core told us which validator this overlay belongs to: keep it, check
+ /// the claims that were waiting for the network id, and tell every peer.
+ async fn set_node_identity(&mut self, identity: NodeIdentity) {
+ info!(
+ "NODE_ID: This overlay belongs to validator {:02x?}...",
+ &identity.node_id[..4]
);
+ let waiting: Vec<(PeerId, (NodeId, [u8; 64]))> = {
+ let mut ids = self.state.identity.lock().unwrap();
+ ids.ours = Some(identity.clone());
+ ids.unverified.drain().collect()
+ };
self.state
- .metrics
- .flood_advertised
- .fetch_add(peers.len() as u64, Ordering::Relaxed);
-
- queue_inv(&self.state, &peers, InvEntry { hash, fee_per_op }).await;
+ .tx_router
+ .lock()
+ .unwrap()
+ .set_local_node(identity.node_id);
+ for (peer, (node_id, signature)) in waiting {
+ accept_node_id_claim(&self.state, peer, node_id, signature).await;
+ }
+ let peers: Vec = self.state.peer_streams.read().await.keys().cloned().collect();
+ for peer in peers {
+ let control = self.control.clone();
+ let identity = identity.clone();
+ tokio::spawn(async move { send_node_identity(control, peer, identity).await });
+ }
+ flush_local_txs(&self.state).await;
}
/// Fetch TX set from a peer - preferring the peer who sent us the SCP message referencing it.
@@ -1402,6 +1675,13 @@ async fn open_streams_to_peer(control: Control, state: Arc, peer_id
}
}
+ // Tell the peer which validator this overlay belongs to, if Core has
+ // said so already (otherwise `set_node_identity` does it later).
+ let identity = state.identity.lock().unwrap().ours.clone();
+ if let Some(identity) = identity {
+ tokio::spawn(send_node_identity(state.control.clone(), peer_id, identity));
+ }
+
// Request SCP state from newly connected peer
info!("Peer {} streams opened, sending SCP state request", peer_id);
let ledger_seq: u32 = 0;
@@ -1650,6 +1930,411 @@ async fn write_framed(stream: &mut Stream, data: &[u8]) -> io::Result<()> {
framed_io::write_frame_parts(stream, &[data]).await
}
+/// Advertise `txs` to every connected peer (the INV/GETDATA flood) and keep
+/// them ready for the demands that follow. Returns false if there was no
+/// peer to advertise them to.
+async fn advertise_txs_to_all(state: &Arc, txs: &[Arc]) -> bool {
+ if txs.is_empty() {
+ return true;
+ }
+ {
+ let mut buffer = state.tx_buffer.write().await;
+ for tx in txs {
+ buffer.insert(Arc::clone(tx));
+ }
+ }
+ let peers: Vec = state.peer_streams.read().await.keys().cloned().collect();
+ if peers.is_empty() {
+ debug!("TX_INV: No peers to announce {} TXs to", txs.len());
+ return false;
+ }
+ debug!(
+ "TX_INV: Announcing {} TXs to {} peers via INV",
+ txs.len(),
+ peers.len()
+ );
+ state
+ .metrics
+ .flood_advertised
+ .fetch_add((peers.len() * txs.len()) as u64, Ordering::Relaxed);
+ for tx in txs {
+ let entry = InvEntry {
+ hash: *tx.hash(),
+ fee_per_op: tx.fee_per_op(),
+ };
+ queue_inv(state, &peers, entry).await;
+ }
+ true
+}
+
+/// Carry out a router plan: bodies straight to the target peers, adverts to
+/// everyone for what has to be flooded.
+async fn execute_send_plan(state: &Arc, plan: SendPlan) {
+ let SendPlan { direct, flood } = plan;
+ if !flood.is_empty() {
+ if advertise_txs_to_all(state, &flood).await {
+ state
+ .metrics
+ .flood_tx_local_flooded
+ .fetch_add(flood.len() as u64, Ordering::Relaxed);
+ } else {
+ // Nobody to tell yet: try again when the targets or the peers
+ // change.
+ let hashes: Vec<[u8; 32]> = flood.iter().map(|tx| *tx.hash()).collect();
+ state.tx_router.lock().unwrap().mark_unflooded(&hashes);
+ }
+ }
+ // Queue the bodies per peer; `direct_send_flush_task` writes them a few
+ // milliseconds later, a full queue is written now.
+ let mut overflow: Vec<(PeerId, Vec<[u8; 32]>)> = Vec::new();
+ let mut full: Vec = Vec::new();
+ {
+ let mut out = state.direct_out.lock().unwrap();
+ for (peer, txs) in direct {
+ let queue = out.queued.entry(peer).or_default();
+ let room = DIRECT_SEND_MAX_QUEUED_TXS.saturating_sub(queue.len());
+ if txs.len() > room {
+ overflow.push((peer, txs[room..].iter().map(|tx| *tx.hash()).collect()));
+ }
+ queue.extend(txs.into_iter().take(room));
+ if queue.len() >= DIRECT_SEND_BATCH_MAX_TXS {
+ full.push(peer);
+ }
+ }
+ }
+ if !overflow.is_empty() {
+ // The peer is not reading: these were not sent. The router offers
+ // them again at a later plan.
+ let mut router = state.tx_router.lock().unwrap();
+ for (peer, hashes) in &overflow {
+ router.mark_unsent(peer, hashes);
+ }
+ }
+ for peer in full {
+ start_direct_sender(state, peer);
+ }
+}
+
+/// Starts the sender task of `peer` unless one is running: it writes what is
+/// queued for the peer, batch after batch, and ends when the queue is empty.
+/// One task per peer, so a peer that stops reading holds one task and a
+/// bounded queue, and a slow peer does not delay another one or the caller.
+fn start_direct_sender(state: &Arc, peer: PeerId) {
+ {
+ let mut out = state.direct_out.lock().unwrap();
+ if out.queued.get(&peer).map_or(true, |q| q.is_empty()) || !out.sending.insert(peer) {
+ return;
+ }
+ }
+ let state = Arc::clone(state);
+ tokio::spawn(async move {
+ loop {
+ let txs = {
+ let mut out = state.direct_out.lock().unwrap();
+ match out.queued.remove(&peer) {
+ Some(txs) if !txs.is_empty() => txs,
+ _ => {
+ out.sending.remove(&peer);
+ return;
+ }
+ }
+ };
+ send_tx_bodies(&state, peer, txs).await;
+ }
+ });
+}
+
+/// Writes the queued direct sends every DIRECT_SEND_BATCH_INTERVAL, and asks
+/// the router every DIRECT_REPLAN_INTERVAL whether anything is owed.
+async fn direct_send_flush_task(state: Arc) {
+ let mut interval = tokio::time::interval(DIRECT_SEND_BATCH_INTERVAL);
+ interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
+ let mut last_replan = Instant::now();
+ loop {
+ interval.tick().await;
+ if last_replan.elapsed() >= DIRECT_REPLAN_INTERVAL {
+ last_replan = Instant::now();
+ flush_local_txs(&state).await;
+ }
+ let peers: Vec = {
+ let out = state.direct_out.lock().unwrap();
+ out.queued
+ .iter()
+ .filter(|(peer, queue)| !queue.is_empty() && !out.sending.contains(*peer))
+ .map(|(peer, _)| *peer)
+ .collect()
+ };
+ for peer in peers {
+ start_direct_sender(&state, peer);
+ }
+ }
+}
+
+/// Ask the router what is owed after the targets or the identified peers
+/// changed, and send it.
+async fn flush_local_txs(state: &Arc) {
+ let plan = state.tx_router.lock().unwrap().plan_all(Instant::now());
+ if plan.is_empty() {
+ return;
+ }
+ debug!(
+ "TX_DIRECT_FLUSH: {} peers get bodies, {} TXs flooded",
+ plan.direct.len(),
+ plan.flood.len()
+ );
+ execute_send_plan(state, plan).await;
+}
+
+/// Send transaction bodies straight to `peer`, without adverts, as batches
+/// of complete TX frames (the framing of a GETDATA response). If a write
+/// fails, the router is told that the remaining ones were not delivered.
+async fn send_tx_bodies(state: &SharedState, peer: PeerId, txs: Vec>) {
+ let mut batch = Vec::new();
+ let mut next = 0;
+ while next < txs.len() {
+ batch.clear();
+ let first = next;
+ let mut payload_bytes = 0;
+ while next < txs.len() {
+ let len = tx_frame_len(&txs[next]);
+ if next > first && batch.len() + len > TX_RESPONSE_BATCH_BYTES {
+ break;
+ }
+ let parts = txs[next].flood_frame_parts();
+ let payload = parts.iter().map(|part| part.len()).sum::();
+ let Ok(prefix) = u32::try_from(payload) else {
+ warn!("TX_DIRECT: transaction too large to frame, skipping");
+ next += 1;
+ continue;
+ };
+ batch.extend_from_slice(&prefix.to_be_bytes());
+ for part in parts {
+ batch.extend_from_slice(part);
+ }
+ payload_bytes += payload;
+ next += 1;
+ }
+ let count = next - first;
+ let frames = framed_io::Frames::Batch(&batch);
+ let result = send_peer_frames(state, peer, StreamType::Tx, frames).await;
+ if record_tx_response_write(state, peer, result, count, payload_bytes) {
+ state
+ .metrics
+ .flood_tx_direct_sent
+ .fetch_add(count as u64, Ordering::Relaxed);
+ state
+ .metrics
+ .flood_tx_direct_sent_bytes
+ .fetch_add(payload_bytes as u64, Ordering::Relaxed);
+ } else {
+ let unsent: Vec<[u8; 32]> = txs[first..].iter().map(|tx| *tx.hash()).collect();
+ state
+ .tx_router
+ .lock()
+ .unwrap()
+ .mark_unsent(&peer, &unsent);
+ return;
+ }
+ }
+}
+
+/// Tell `peer` which validator this overlay belongs to, and wait for it to
+/// acknowledge the frame. Retried a few times: the stream library drops an
+/// inbound stream when several arrive faster than they are accepted (every
+/// peer identifies itself at the same moment when a network starts), and a
+/// dropped stream is not always an error on the writing side.
+async fn send_node_identity(control: Control, peer: PeerId, identity: NodeIdentity) {
+ const ATTEMPTS: u32 = 12;
+ const MAX_DELAY: Duration = Duration::from_secs(5);
+ let mut delay = Duration::from_millis(100);
+ for attempt in 1..=ATTEMPTS {
+ match try_send_node_identity(control.clone(), peer, &identity).await {
+ Ok(()) => return,
+ Err(e) if e.kind() == io::ErrorKind::Unsupported => {
+ // A peer without this protocol never identifies itself
+ // either; transactions are flooded when it is a target.
+ debug!("NODE_ID: {} does not take node ids: {}", peer, e);
+ return;
+ }
+ Err(e) => {
+ debug!(
+ "NODE_ID: Sending the node id to {} failed (attempt {}): {}",
+ peer, attempt, e
+ );
+ }
+ }
+ tokio::time::sleep(delay).await;
+ delay = (delay * 2).min(MAX_DELAY);
+ }
+ // About half a minute of trying. If the connection is still up, this
+ // node floods its transactions whenever that validator is a target.
+ warn!(
+ "NODE_ID: Could not tell {} which validator this overlay belongs to",
+ peer
+ );
+}
+
+/// One attempt of [`send_node_identity`].
+async fn try_send_node_identity(
+ mut control: Control,
+ peer: PeerId,
+ identity: &NodeIdentity,
+) -> io::Result<()> {
+ use futures::AsyncWriteExt;
+ let mut stream = control
+ .open_stream(peer, NODE_ID_PROTOCOL)
+ .await
+ .map_err(|e| match e {
+ libp2p_stream::OpenStreamError::UnsupportedProtocol(_) => {
+ io::Error::new(io::ErrorKind::Unsupported, e)
+ }
+ other => io::Error::new(io::ErrorKind::NotConnected, other),
+ })?;
+ let mut frame = [0u8; NODE_ID_FRAME_LEN];
+ frame[..32].copy_from_slice(&identity.node_id);
+ frame[32..].copy_from_slice(&identity.signature);
+ write_framed(&mut stream, &frame).await?;
+ let mut ack = [0u8; 1];
+ tokio::time::timeout(NODE_ID_READ_TIMEOUT, stream.read_exact(&mut ack))
+ .await
+ .map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "no acknowledgement"))??;
+ let _ = stream.close().await;
+ if ack[0] != NODE_ID_ACK_TAKEN {
+ // The peer read the frame but could not use it yet (it had not
+ // registered this connection): send it again.
+ return Err(io::Error::new(io::ErrorKind::WouldBlock, "not taken yet"));
+ }
+ Ok(())
+}
+
+/// Peers telling us which validator they belong to.
+async fn handle_inbound_node_id_streams(mut incoming: IncomingStreams, state: Arc) {
+ while let Some((peer_id, mut stream)) = incoming.next().await {
+ let state = state.clone();
+ tokio::spawn(async move {
+ let read = tokio::time::timeout(
+ NODE_ID_READ_TIMEOUT,
+ read_framed_with_limit(&mut stream, NODE_ID_FRAME_LEN),
+ );
+ let frame = match read.await {
+ Ok(Ok(frame)) if frame.len() == NODE_ID_FRAME_LEN => frame,
+ Ok(Ok(frame)) => {
+ debug!(
+ "NODE_ID: Ignoring a node id frame of {} bytes from {}",
+ frame.len(),
+ peer_id
+ );
+ return;
+ }
+ Ok(Err(e)) => {
+ debug!("NODE_ID: Node id stream from {} failed: {}", peer_id, e);
+ return;
+ }
+ Err(_) => {
+ debug!("NODE_ID: Node id stream from {} sent nothing", peer_id);
+ return;
+ }
+ };
+ let mut node_id = [0u8; 32];
+ node_id.copy_from_slice(&frame[..32]);
+ let mut signature = [0u8; 64];
+ signature.copy_from_slice(&frame[32..]);
+ let taken = accept_node_id_claim(&state, peer_id, node_id, signature).await;
+ {
+ use futures::AsyncWriteExt;
+ let ack = if taken {
+ NODE_ID_ACK_TAKEN
+ } else {
+ NODE_ID_ACK_RETRY
+ };
+ let _ = stream.write_all(&[ack]).await;
+ let _ = stream.flush().await;
+ }
+ });
+ }
+}
+
+/// A peer claims to be validator `node_id`. Accept it if the signature
+/// binds that validator to the peer id of this connection; keep it for later
+/// if Core has not told us the network id yet.
+/// Returns false if the claim could not be used yet and should be repeated.
+async fn accept_node_id_claim(
+ state: &Arc,
+ peer: PeerId,
+ node_id: NodeId,
+ signature: [u8; 64],
+) -> bool {
+ // Nothing to do for a claim that is already accepted (a peer repeats it
+ // on every connection it establishes).
+ if state.tx_router.lock().unwrap().node_of(&peer) == Some(&node_id) {
+ return true;
+ }
+ // Only a connected peer can be a target. The peer's stream can get here
+ // before this side has registered the connection: the sender repeats the
+ // claim.
+ if !state.peer_streams.read().await.contains_key(&peer) {
+ return false;
+ }
+ let network_id = {
+ let mut ids = state.identity.lock().unwrap();
+ let known = ids.ours.as_ref().map(|ours| ours.network_id);
+ match known {
+ Some(network_id) => {
+ // A signature check, a log line and a pass over the pending
+ // transactions per claim: bound what one connection can ask
+ // for. (A waiting claim is counted when it is checked.)
+ let claims = ids.claims.entry(peer).or_insert(0);
+ if *claims >= MAX_NODE_ID_CLAIMS_PER_PEER {
+ return true;
+ }
+ *claims += 1;
+ network_id
+ }
+ None => {
+ ids.unverified.insert(peer, (node_id, signature));
+ return true;
+ }
+ }
+ };
+ if !verify_node_id_binding(&network_id, &peer, &node_id, &signature) {
+ state
+ .metrics
+ .node_id_rejected
+ .fetch_add(1, Ordering::Relaxed);
+ warn!(
+ "NODE_ID_REJECT: Peer {} claims validator {:02x?}... with a signature that does not bind it to this connection",
+ peer,
+ &node_id[..4]
+ );
+ return true;
+ }
+ info!(
+ "NODE_ID: Peer {} is validator {:02x?}...",
+ peer,
+ &node_id[..4]
+ );
+ let mut identified = {
+ let mut router = state.tx_router.lock().unwrap();
+ router.peer_identified(peer, node_id);
+ router.identified_peer_count()
+ };
+ // The connection may have closed between the check above and now; its
+ // close handler has then already run and would not undo this.
+ if !state.peer_streams.read().await.contains_key(&peer) {
+ let mut router = state.tx_router.lock().unwrap();
+ router.peer_disconnected(&peer);
+ identified = router.identified_peer_count();
+ state.identity.lock().unwrap().claims.remove(&peer);
+ }
+ state
+ .metrics
+ .node_id_identified
+ .store(identified as i64, Ordering::Relaxed);
+ // The peer may be a target that could not be reached until now.
+ flush_local_txs(state).await;
+ true
+}
+
/// Flush INV batch for a specific peer
async fn flush_inv_batch_to_peer(state: &Arc, peer: PeerId) {
let batch = {
@@ -1960,6 +2645,51 @@ async fn handle_inv_batch(state: &Arc, peer_id: &PeerId, batch: Inv
peer_id
);
+ // A transaction this node got as a direct send (and so did not relay)
+ // is now being advertised: its origin has fallen back to flooding, which
+ // has to pass through this node too. Advertise it to the other peers,
+ // once.
+ let join_flood: Vec> = {
+ let mut router = state.tx_router.lock().unwrap();
+ if router.has_direct() {
+ batch
+ .entries
+ .iter()
+ .filter_map(|entry| router.join_flood(&entry.hash))
+ .collect()
+ } else {
+ Vec::new()
+ }
+ };
+ if !join_flood.is_empty() {
+ // Ready for the demands that follow the adverts.
+ {
+ let mut buffer = state.tx_buffer.write().await;
+ for tx in &join_flood {
+ buffer.insert(Arc::clone(tx));
+ }
+ }
+ let peers: Vec = state
+ .peer_streams
+ .read()
+ .await
+ .keys()
+ .filter(|p| *p != peer_id)
+ .cloned()
+ .collect();
+ debug!(
+ "TX_RELAY: Joining the flood of {} TXs received directly earlier",
+ join_flood.len()
+ );
+ for tx in join_flood {
+ let entry = InvEntry {
+ hash: *tx.hash(),
+ fee_per_op: tx.fee_per_op(),
+ };
+ queue_inv(state, &peers, entry).await;
+ }
+ }
+
// Each structure is locked once per batch rather than once per entry, and
// the two locks are never held together. Entries keep their batch order.
// Skip TXs we already have; they are not recorded as sources.
@@ -2093,7 +2823,14 @@ async fn serve_getdata(state: &SharedState, peer: PeerId, hashes: Vec<[u8; 32]>)
if bytes < TX_RESPONSE_BATCH_BYTES {
let mut buffer = state.tx_buffer.write().await;
for hash in hashes.by_ref() {
- let Some(tx) = buffer.get(&hash) else {
+ // The buffer keeps a flooded transaction for a short time
+ // only. One this node accepted itself, or received directly,
+ // is kept until it is externalized, and may be demanded long
+ // after it arrived (a flood that starts later).
+ let found = buffer
+ .get(&hash)
+ .or_else(|| state.tx_router.lock().unwrap().body(&hash));
+ let Some(tx) = found else {
state
.metrics
.flood_unfulfilled_unknown
@@ -2234,9 +2971,22 @@ async fn handle_tx_response(state: &Arc, peer_id: &PeerId, tx: Arc<
.metrics
.flood_unique_recv
.fetch_add(tx_len, Ordering::Relaxed);
+ // A first copy of a transaction that one of the last ledgers already
+ // has. It is in the mempool now, and stays until a build finds it
+ // invalid (or, where ledgers are not applied, includes it again).
+ if state.tx_router.lock().unwrap().was_finalized(&hash) {
+ state.metrics.flood_tx_late_copy.fetch_add(1, Ordering::Relaxed);
+ debug!(
+ "TX_LATE_COPY: TX {:02x?}... from {} arrived after it externalized",
+ &hash[..4],
+ peer_id
+ );
+ }
- // Remove from pending requests and measure pull latency
- {
+ // Remove from pending requests and measure pull latency. A body this
+ // node demanded arrived over the flood; one it did not ask for was sent
+ // directly by a node that is sending to its targets only.
+ let solicited = {
let mut pending = state.pending_getdata.write().await;
if let Some(req) = pending.remove(&hash) {
let pull_us = req.first_sent_at.elapsed().as_micros() as u64;
@@ -2248,8 +2998,11 @@ async fn handle_tx_response(state: &Arc, peer_id: &PeerId, tx: Arc<
.metrics
.flood_tx_pull_latency_count
.fetch_add(1, Ordering::Relaxed);
+ true
+ } else {
+ false
}
- }
+ };
// Store in buffer for responding to others' GETDATA
{
@@ -2264,6 +3017,24 @@ async fn handle_tx_response(state: &Arc, peer_id: &PeerId, tx: Arc<
peer_id
);
+ if !solicited {
+ // A direct send: the sender delivers it to every target itself, so
+ // it is kept for this node's own tx sets and not relayed. Remember
+ // it in case its origin floods it later (see `handle_inv_batch`).
+ state
+ .metrics
+ .flood_tx_direct_recv
+ .fetch_add(1, Ordering::Relaxed);
+ state.tx_router.lock().unwrap().note_direct(Arc::clone(&tx));
+ debug!(
+ "TX_DIRECT_RECV: TX {:02x?}... from {} kept, not relayed",
+ &hash[..4],
+ peer_id
+ );
+ record_recv_transaction(state, recv_start);
+ return;
+ }
+
// RELAY: Announce to other peers via INV
let peers_to_announce: Vec = {
let streams = state.peer_streams.read().await;
@@ -2292,7 +3063,11 @@ async fn handle_tx_response(state: &Arc, peer_id: &PeerId, tx: Arc<
queue_inv(state, &peers_to_announce, InvEntry { hash, fee_per_op }).await;
}
- // Record recv-transaction timing
+ record_recv_transaction(state, recv_start);
+}
+
+/// Record recv-transaction timing
+fn record_recv_transaction(state: &SharedState, recv_start: std::time::Instant) {
let elapsed_us = recv_start.elapsed().as_micros() as u64;
state
.metrics
@@ -2646,6 +3421,9 @@ mod quic_window_experiment;
#[cfg(test)]
mod txset_push_tests;
+#[cfg(test)]
+mod tx_direct_tests;
+
#[cfg(test)]
mod tests {
use super::*;
diff --git a/overlay/src/libp2p_overlay/dispatch_tests.rs b/overlay/src/libp2p_overlay/dispatch_tests.rs
index 361001c85..b9d201f9b 100644
--- a/overlay/src/libp2p_overlay/dispatch_tests.rs
+++ b/overlay/src/libp2p_overlay/dispatch_tests.rs
@@ -514,6 +514,24 @@ async fn received_tx_is_announced_to_every_peer_except_known_sources() {
.await
.record_source(*tx.hash(), advertiser);
+ // A body nobody here demanded was sent directly by a node that delivers
+ // to its targets itself: it is kept, and announced to nobody.
+ let direct = test_tx(2);
+ handle_tx_response(&state, &source, Arc::clone(&direct)).await;
+ assert!(state.tx_seen.read().await.contains(direct.hash()));
+ {
+ let mut batcher = state.inv_batcher.write().await;
+ for peer in &others[1..] {
+ assert!(batcher.flush(peer).is_none(), "direct send was relayed");
+ }
+ }
+
+ // A body this node demanded (the flood) is relayed.
+ state
+ .pending_getdata
+ .write()
+ .await
+ .insert(*tx.hash(), source);
handle_tx_response(&state, &source, Arc::clone(&tx)).await;
let mut batcher = state.inv_batcher.write().await;
@@ -562,7 +580,15 @@ async fn inv_batches_completed_by_relay_or_local_broadcast_reach_the_peer() {
}
let tx = test_tx(i64::from(round) + 1);
if round == 0 {
- handle_tx_response(&sender.state, &PeerId::random(), tx).await;
+ // A relay: the body answers this node's own demand.
+ let source = PeerId::random();
+ sender
+ .state
+ .pending_getdata
+ .write()
+ .await
+ .insert(*tx.hash(), source);
+ handle_tx_response(&sender.state, &source, tx).await;
} else {
sender.handle.broadcast_tx(tx).await;
}
diff --git a/overlay/src/libp2p_overlay/tx_direct_tests.rs b/overlay/src/libp2p_overlay/tx_direct_tests.rs
new file mode 100644
index 000000000..88de735d3
--- /dev/null
+++ b/overlay/src/libp2p_overlay/tx_direct_tests.rs
@@ -0,0 +1,426 @@
+//! Tests for sending transactions to the target validators only, and for
+//! the identity binding that ties a peer connection to a validator.
+
+use super::*;
+use libp2p::identity::ed25519;
+
+const NETWORK_ID: [u8; 32] = [7; 32];
+
+/// One overlay node on loopback with a validator key of its own.
+struct Node {
+ handle: OverlayHandle,
+ state: Arc,
+ metrics: Arc,
+ peer_id: PeerId,
+ port: u16,
+ validator: ed25519::Keypair,
+ admissions: mpsc::UnboundedReceiver,
+ /// Transactions read from `admissions` so far.
+ admitted: HashSet<[u8; 32]>,
+ _events: mpsc::UnboundedReceiver,
+}
+
+impl Node {
+ fn node_id(&self) -> NodeId {
+ self.validator.public().to_bytes()
+ }
+
+ /// What Core sends in `SetNodeIdentity`: this node's validator key
+ /// signing the binding to this overlay's peer id.
+ fn identity(&self) -> NodeIdentity {
+ sign_identity(&self.validator, &NETWORK_ID, &self.peer_id)
+ }
+
+ async fn identify(&self) {
+ self.handle.set_node_identity(self.identity()).await;
+ }
+
+ fn identified_peers(&self) -> usize {
+ self.state.tx_router.lock().unwrap().identified_peer_count()
+ }
+
+ /// Core accepted `tx`.
+ async fn submit(&self, tx: &Arc) {
+ let plan = self.handle.add_local_tx(Arc::clone(tx));
+ self.handle.send_local_tx(plan).await;
+ }
+
+ async fn set_targets(&self, targets: Option>) {
+ if self.handle.set_tx_targets(targets) {
+ self.handle.flush_local_txs().await;
+ }
+ }
+
+ /// Transactions newly admitted to this node's mempool within `window`.
+ async fn admitted_within(&mut self, window: Duration) -> Vec<[u8; 32]> {
+ let deadline = tokio::time::Instant::now() + window;
+ let mut hashes = Vec::new();
+ while let Ok(Some(command)) = tokio::time::timeout_at(deadline, self.admissions.recv()).await
+ {
+ if let CoreCommand::SubmitTx { tx, .. } = command {
+ self.admitted.insert(*tx.hash());
+ hashes.push(*tx.hash());
+ }
+ }
+ hashes
+ }
+
+ /// Wait until `tx` has been admitted to this node's mempool (in any
+ /// order relative to other transactions).
+ async fn expect_admitted(&mut self, tx: &Arc) {
+ let deadline = tokio::time::Instant::now() + Duration::from_secs(10);
+ while !self.admitted.contains(tx.hash()) {
+ match tokio::time::timeout_at(deadline, self.admissions.recv()).await {
+ Ok(Some(CoreCommand::SubmitTx { tx: got, .. })) => {
+ if got.hash() == tx.hash() {
+ assert_eq!(got.bytes(), tx.bytes());
+ }
+ self.admitted.insert(*got.hash());
+ }
+ Ok(Some(_)) => {}
+ _ => panic!("transaction {:02x?} was not admitted", &tx.hash()[..4]),
+ }
+ }
+ }
+
+ fn counter(&self, pick: impl Fn(&OverlayMetrics) -> &AtomicU64) -> u64 {
+ pick(&self.metrics).load(Ordering::Relaxed)
+ }
+}
+
+fn sign_identity(validator: &ed25519::Keypair, network_id: &[u8; 32], peer: &PeerId) -> NodeIdentity {
+ let signature: [u8; 64] = validator
+ .sign(&node_id_binding(network_id, peer))
+ .try_into()
+ .unwrap();
+ NodeIdentity {
+ network_id: *network_id,
+ node_id: validator.public().to_bytes(),
+ signature,
+ }
+}
+
+async fn start_node(port: u16) -> Node {
+ let keypair = Keypair::generate_ed25519();
+ let peer_id = keypair.public().to_peer_id();
+ let metrics = Arc::new(OverlayMetrics::new());
+ let (handle, events, admissions, overlay) =
+ create_test_overlay(keypair, Arc::clone(&metrics)).unwrap();
+ let state = Arc::clone(&overlay.state);
+ tokio::spawn(async move { overlay.run("127.0.0.1", port).await });
+ tokio::time::sleep(Duration::from_millis(100)).await;
+ Node {
+ handle,
+ state,
+ metrics,
+ peer_id,
+ port,
+ validator: ed25519::Keypair::generate(),
+ admissions,
+ admitted: HashSet::new(),
+ _events: events,
+ }
+}
+
+/// Dial `listener` from `dialer` and wait until both sides have streams.
+async fn connect(dialer: &Node, listener: &Node) {
+ let addr: Multiaddr = format!("/ip4/127.0.0.1/udp/{}/quic-v1", listener.port)
+ .parse()
+ .unwrap();
+ dialer.handle.dial(addr).await;
+ let deadline = Instant::now() + Duration::from_secs(10);
+ loop {
+ let a = dialer
+ .state
+ .peer_streams
+ .read()
+ .await
+ .contains_key(&listener.peer_id);
+ let b = listener
+ .state
+ .peer_streams
+ .read()
+ .await
+ .contains_key(&dialer.peer_id);
+ if a && b {
+ break;
+ }
+ assert!(Instant::now() < deadline, "peers did not connect");
+ tokio::time::sleep(Duration::from_millis(20)).await;
+ }
+ // Let the SCP state exchange of a new connection settle.
+ tokio::time::sleep(Duration::from_millis(200)).await;
+}
+
+async fn wait_until(what: &str, condition: impl Fn() -> bool) {
+ let deadline = Instant::now() + Duration::from_secs(10);
+ while !condition() {
+ assert!(Instant::now() < deadline, "timed out waiting for {what}");
+ tokio::time::sleep(Duration::from_millis(20)).await;
+ }
+}
+
+/// The key scenario: in a full mesh a transaction reaches the two targets
+/// and nobody else, without a single advert; a pending transaction follows
+/// the targets; an externalized one is not sent again.
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn a_transaction_goes_to_the_targets_only_and_follows_them() {
+ let origin = start_node(19951).await;
+ let mut leader1 = start_node(19952).await;
+ let mut leader2 = start_node(19953).await;
+ let mut other = start_node(19954).await;
+ connect(&origin, &leader1).await;
+ connect(&origin, &leader2).await;
+ connect(&origin, &other).await;
+ connect(&leader1, &leader2).await;
+ connect(&leader1, &other).await;
+ connect(&leader2, &other).await;
+
+ // Identities arrive one node at a time, so some claims reach a node
+ // before its own identity (and with it the network id) does.
+ for node in [&leader1, &leader2, &other, &origin] {
+ node.identify().await;
+ }
+ for node in [&origin, &leader1, &leader2, &other] {
+ wait_until("every peer to be identified", || {
+ node.identified_peers() == 3
+ })
+ .await;
+ }
+ {
+ let router = origin.state.tx_router.lock().unwrap();
+ assert_eq!(router.node_of(&leader1.peer_id), Some(&leader1.node_id()));
+ assert_eq!(router.node_of(&other.peer_id), Some(&other.node_id()));
+ }
+
+ origin
+ .set_targets(Some(vec![leader1.node_id(), leader2.node_id()]))
+ .await;
+ let included = test_tx(1);
+ let pending = test_tx(2);
+ origin.submit(&included).await;
+ origin.submit(&pending).await;
+
+ leader1.expect_admitted(&included).await;
+ leader1.expect_admitted(&pending).await;
+ leader2.expect_admitted(&included).await;
+ leader2.expect_admitted(&pending).await;
+ // Nobody else gets them: the origin sent no advert, and the targets do
+ // not relay what was sent to them directly.
+ assert!(other.admitted_within(Duration::from_millis(700)).await.is_empty());
+ assert_eq!(origin.counter(|m| &m.flood_advertised), 0);
+ assert_eq!(origin.counter(|m| &m.flood_tx_direct_sent), 4);
+ assert_eq!(origin.counter(|m| &m.flood_tx_local_flooded), 0);
+ for leader in [&leader1, &leader2] {
+ assert_eq!(leader.counter(|m| &m.flood_tx_direct_recv), 2);
+ assert_eq!(leader.counter(|m| &m.flood_advertised), 0);
+ assert_eq!(leader.counter(|m| &m.flood_demanded), 0);
+ }
+
+ // `included` externalizes; the next ledger's leaders are `other` and
+ // `leader1`. `pending` is not new any more and may be in the set the
+ // old targets are voting on: it does not move yet.
+ origin.handle.ledger_externalized(&[*included.hash()]);
+ assert_eq!(origin.handle.local_tx_count(), 1);
+ origin
+ .set_targets(Some(vec![other.node_id(), leader1.node_id()]))
+ .await;
+ assert!(other.admitted_within(Duration::from_millis(300)).await.is_empty());
+ assert_eq!(origin.counter(|m| &m.flood_tx_direct_sent), 4);
+ // The next two ledgers do not have it either: now it moves, and only
+ // to the new target.
+ origin.handle.ledger_externalized(&[]);
+ origin.handle.ledger_externalized(&[]);
+ origin.handle.flush_local_txs().await;
+ other.expect_admitted(&pending).await;
+ assert!(other.admitted_within(Duration::from_millis(500)).await.is_empty());
+ assert_eq!(origin.counter(|m| &m.flood_tx_direct_sent), 5);
+ assert_eq!(leader1.counter(|m| &m.flood_tx_direct_recv), 2);
+ assert_eq!(origin.counter(|m| &m.flood_advertised), 0);
+
+ // A copy that reaches a node after its transaction externalized there is
+ // admitted all the same (the mempool does not know), and counted.
+ let late = test_tx(3);
+ other.handle.ledger_externalized(&[*late.hash()]);
+ assert_eq!(other.counter(|m| &m.flood_tx_late_copy), 0);
+ origin.submit(&late).await;
+ other.expect_admitted(&late).await;
+ wait_until("the late copy to be counted", || {
+ other.counter(|m| &m.flood_tx_late_copy) == 1
+ })
+ .await;
+ assert_eq!(leader1.counter(|m| &m.flood_tx_late_copy), 0);
+}
+
+/// Without a target list, and when a target is not an identified peer, a
+/// transaction is flooded as before and reaches every node.
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn without_reachable_targets_a_transaction_is_flooded() {
+ let origin = start_node(19956).await;
+ let mut relay = start_node(19957).await;
+ let mut far = start_node(19958).await;
+ // A chain: the origin is not connected to `far`.
+ connect(&origin, &relay).await;
+ connect(&relay, &far).await;
+ for node in [&origin, &relay, &far] {
+ node.identify().await;
+ }
+ wait_until("the origin to identify its peer", || {
+ origin.identified_peers() == 1
+ })
+ .await;
+
+ // No targets yet: flood.
+ let first = test_tx(1);
+ origin.submit(&first).await;
+ relay.expect_admitted(&first).await;
+ far.expect_admitted(&first).await;
+ assert_eq!(origin.counter(|m| &m.flood_tx_local_flooded), 1);
+ assert_eq!(origin.counter(|m| &m.flood_tx_direct_sent), 0);
+
+ // A target the origin has no connection to: flood, through the relay.
+ origin
+ .set_targets(Some(vec![relay.node_id(), far.node_id()]))
+ .await;
+ let second = test_tx(2);
+ origin.submit(&second).await;
+ relay.expect_admitted(&second).await;
+ far.expect_admitted(&second).await;
+ assert_eq!(origin.counter(|m| &m.flood_tx_local_flooded), 2);
+ assert_eq!(origin.counter(|m| &m.flood_tx_direct_sent), 0);
+}
+
+/// A node that got a transaction directly does not relay it, but joins in
+/// when the origin later has to flood it (a new target it cannot reach).
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn a_directly_received_transaction_joins_a_later_flood() {
+ let origin = start_node(19961).await;
+ let mut relay = start_node(19962).await;
+ let mut far = start_node(19963).await;
+ connect(&origin, &relay).await;
+ connect(&relay, &far).await;
+ for node in [&origin, &relay, &far] {
+ node.identify().await;
+ }
+ wait_until("the origin to identify its peer", || {
+ origin.identified_peers() == 1
+ })
+ .await;
+
+ origin.set_targets(Some(vec![relay.node_id()])).await;
+ let tx = test_tx(1);
+ origin.submit(&tx).await;
+ relay.expect_admitted(&tx).await;
+ assert!(far.admitted_within(Duration::from_millis(700)).await.is_empty());
+ assert_eq!(relay.counter(|m| &m.flood_tx_direct_recv), 1);
+
+ // Time passes: the transaction is long gone from the short-lived buffers
+ // that serve a flood's demands.
+ *origin.state.tx_buffer.write().await = TxBuffer::new();
+ *relay.state.tx_buffer.write().await = TxBuffer::new();
+
+ // The next leader is `far`, which the origin cannot reach: it floods
+ // the pending transaction, and the relay, which already has it, passes
+ // the advert on and serves the body.
+ origin.set_targets(Some(vec![far.node_id()])).await;
+ far.expect_admitted(&tx).await;
+ assert_eq!(origin.counter(|m| &m.flood_tx_local_flooded), 1);
+}
+
+/// An identity claim only counts if the validator's key signed it for this
+/// very connection.
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn a_forged_identity_claim_is_rejected() {
+ let honest = start_node(19966).await;
+ let mut forger = start_node(19967).await;
+ let mut replayer = start_node(19968).await;
+ let victim = ed25519::Keypair::generate();
+ connect(&forger, &honest).await;
+ connect(&replayer, &honest).await;
+ honest.identify().await;
+
+ // Claims the victim's node id, signed with the forger's own key.
+ let mut forged = sign_identity(&forger.validator, &NETWORK_ID, &forger.peer_id);
+ forged.node_id = victim.public().to_bytes();
+ forger.handle.set_node_identity(forged).await;
+ // Replays a binding the victim signed for another peer id.
+ let replayed = sign_identity(&victim, &NETWORK_ID, &forger.peer_id);
+ replayer.handle.set_node_identity(replayed).await;
+
+ wait_until("both claims to be rejected", || {
+ honest.counter(|m| &m.node_id_rejected) == 2
+ })
+ .await;
+ assert_eq!(honest.identified_peers(), 0);
+
+ // A binding for another network does not count either.
+ let other_network = sign_identity(&victim, &[8; 32], &replayer.peer_id);
+ replayer
+ .handle
+ .set_node_identity(NodeIdentity {
+ network_id: NETWORK_ID,
+ ..other_network
+ })
+ .await;
+ wait_until("the third claim to be rejected", || {
+ honest.counter(|m| &m.node_id_rejected) == 3
+ })
+ .await;
+ assert_eq!(honest.identified_peers(), 0);
+
+ // A peer cannot keep this node busy with claims: beyond a few per
+ // connection they are not even checked.
+ for _ in 0..(MAX_NODE_ID_CLAIMS_PER_PEER + 4) {
+ let forged = sign_identity(&forger.validator, &NETWORK_ID, &replayer.peer_id);
+ forger.handle.set_node_identity(forged).await;
+ }
+ tokio::time::sleep(Duration::from_millis(700)).await;
+ assert_eq!(
+ honest.counter(|m| &m.node_id_rejected),
+ 2 + u64::from(MAX_NODE_ID_CLAIMS_PER_PEER),
+ "the three before (one of them the forger's), and the forger's up to its limit"
+ );
+ assert_eq!(honest.identified_peers(), 0);
+
+ // With the victim as the target, nothing is sent directly to either
+ // impostor: the transaction is flooded instead.
+ honest
+ .set_targets(Some(vec![victim.public().to_bytes()]))
+ .await;
+ let tx = test_tx(1);
+ honest.submit(&tx).await;
+ forger.expect_admitted(&tx).await;
+ replayer.expect_admitted(&tx).await;
+ assert_eq!(honest.counter(|m| &m.flood_tx_direct_sent), 0);
+ assert_eq!(honest.counter(|m| &m.flood_tx_local_flooded), 1);
+}
+
+#[test]
+fn the_binding_depends_on_network_and_peer() {
+ let peer_a = Keypair::generate_ed25519().public().to_peer_id();
+ let peer_b = Keypair::generate_ed25519().public().to_peer_id();
+ let validator = ed25519::Keypair::generate();
+ let identity = sign_identity(&validator, &NETWORK_ID, &peer_a);
+ assert!(verify_node_id_binding(
+ &NETWORK_ID,
+ &peer_a,
+ &identity.node_id,
+ &identity.signature
+ ));
+ assert!(!verify_node_id_binding(
+ &NETWORK_ID,
+ &peer_b,
+ &identity.node_id,
+ &identity.signature
+ ));
+ assert!(!verify_node_id_binding(
+ &[8; 32],
+ &peer_a,
+ &identity.node_id,
+ &identity.signature
+ ));
+ assert_ne!(
+ node_id_binding(&NETWORK_ID, &peer_a),
+ node_id_binding(&NETWORK_ID, &peer_b)
+ );
+}
diff --git a/overlay/src/main.rs b/overlay/src/main.rs
index 55d4dc1a2..7b542a118 100644
--- a/overlay/src/main.rs
+++ b/overlay/src/main.rs
@@ -26,7 +26,8 @@ use stellar_overlay::flood::{
use stellar_overlay::integrated::{Overlay, OverlayHandle};
use stellar_overlay::ipc::{CoreIpc, Message, MessageType};
use stellar_overlay::libp2p_overlay::{
- create_overlay, OverlayEvent as LibP2pOverlayEvent, OverlayHandle as LibP2pOverlayHandle,
+ create_overlay, NodeIdentity, OverlayEvent as LibP2pOverlayEvent,
+ OverlayHandle as LibP2pOverlayHandle,
};
use stellar_overlay::metrics::OverlayMetrics;
use stellar_overlay::wire::ValidatedTx;
@@ -560,6 +561,17 @@ impl App {
async fn run(mut self) {
info!("Overlay started, processing Core messages");
+ // Ask Core to bind this overlay's peer id to the validator's key;
+ // it answers with `SetNodeIdentity`.
+ let peer_id = self.libp2p_handle.local_peer_id().to_bytes();
+ if let Err(e) = self
+ .core_ipc
+ .sender
+ .send(Message::new(MessageType::OverlayIdentity, peer_id))
+ {
+ error!("Failed to send the overlay identity to Core: {}", e);
+ }
+
// Safety-net reconnect timer: re-dial all configured peers every 30s.
// Uses PeerId-based dials for known peers (libp2p skips if already connected).
// Falls back to address-based dials for peers we haven't connected to yet.
@@ -1422,13 +1434,62 @@ impl App {
// Add to mempool
self.overlay_handle.submit_tx(Arc::clone(&tx));
- // Broadcast TX via libp2p QUIC (dedicated stream)
+ // Core accepted this transaction from a client: this node
+ // keeps it and sends it until it is externalized, to the
+ // targets Core named or, without targets, to every peer.
+ // The router is updated here, in the order of Core's
+ // messages; only the sending is left to a task.
+ let plan = self.libp2p_handle.add_local_tx(tx);
+ let handle = self.libp2p_handle.clone();
+ tokio::spawn(async move {
+ handle.send_local_tx(plan).await;
+ });
+ }
+
+ MessageType::SetNodeIdentity => {
+ // Payload: [networkID:32][nodeID:32][signature:64]
+ if msg.payload.len() != 128 {
+ warn!(
+ "SetNodeIdentity payload has {} bytes, expected 128",
+ msg.payload.len()
+ );
+ return true;
+ }
+ let mut identity = NodeIdentity {
+ network_id: [0; 32],
+ node_id: [0; 32],
+ signature: [0; 64],
+ };
+ identity.network_id.copy_from_slice(&msg.payload[0..32]);
+ identity.node_id.copy_from_slice(&msg.payload[32..64]);
+ identity.signature.copy_from_slice(&msg.payload[64..128]);
let handle = self.libp2p_handle.clone();
tokio::spawn(async move {
- handle.broadcast_tx(tx).await;
+ handle.set_node_identity(identity).await;
});
}
+ MessageType::SetTxTargets => {
+ // Payload: [everyone:1][count:4][nodeID:32]...
+ let Some(targets) = parse_tx_targets(&msg.payload) else {
+ warn!(
+ "SetTxTargets payload malformed ({} bytes)",
+ msg.payload.len()
+ );
+ return true;
+ };
+ match &targets {
+ Some(nodes) => debug!("TX_TARGETS: {} validators get transactions", nodes.len()),
+ None => debug!("TX_TARGETS: every peer gets transactions"),
+ }
+ if self.libp2p_handle.set_tx_targets(targets) {
+ let handle = self.libp2p_handle.clone();
+ tokio::spawn(async move {
+ handle.flush_local_txs().await;
+ });
+ }
+ }
+
MessageType::RequestScpState => {
// Core is asking us to request SCP state from peers
// Payload is ledger sequence (u32, 4 bytes)
@@ -1520,6 +1581,21 @@ impl App {
}
}
+ // Nothing is left to send for them. Core uses this
+ // message with a zero set hash to drop transactions for
+ // its own reasons; only a real set is a ledger.
+ if tx_set_hash == [0u8; 32] {
+ self.libp2p_handle.forget_local_txs(&tx_hashes);
+ } else {
+ // Transactions this ledger did not have may now be
+ // due at targets they were held back from.
+ self.libp2p_handle.ledger_externalized(&tx_hashes);
+ let handle = self.libp2p_handle.clone();
+ tokio::spawn(async move {
+ handle.flush_local_txs().await;
+ });
+ }
+
// Remove TXs from mempool and WAIT for completion
// This prevents race where next nomination queries stale mempool
if !tx_hashes.is_empty() {
@@ -1732,6 +1808,31 @@ impl App {
}
}
+/// Parse a `SetTxTargets` payload: `[everyone:1][count:4][nodeID:32]...`.
+/// Returns `None` if it is malformed, `Some(None)` for "every peer".
+fn parse_tx_targets(payload: &[u8]) -> Option