Repository navigation
Level 0 of the bucket list as parallel shards under a commutative hash (experiment 8, prototype) - #5510
Draft
SirTyson wants to merge 28 commits into
Draft
Level 0 of the bucket list as parallel shards under a commutative hash (experiment 8, prototype)#5510SirTyson wants to merge 28 commits into
SirTyson wants to merge 28 commits into
Conversation
…t is settable (completiontimeout) A generator of the real-apply load reported failure although every one of its payments was applied: after its last submission loadgen waits 20 ledgers for its accounts, and on a 90-validator network 1 to 9 of 90 generators had a last batch that was included 25 to 70 s later (payments go to the round leaders only, without an acknowledgement). The run then got no verdict on inclusion. SAC_PAYMENT now waits 150 ledgers by default, and generateload takes completiontimeout=<ledgers> for any mode. Other modes keep 20.
A lookup in the per-bucket account cache (BUCKETLIST_DB_MEMORY_FOR_CACHING) took the cache's lock exclusively even for a hit, because RandomEvictionCache::maybeGet updates the entry's access time and the hit/miss counters. Ledger apply loads accounts from many worker threads at once, and every load asks the cache of each disk-indexed bucket on its way down the bucket list, so with the cache on the workers' account loads queued up on that lock (measured on a 6-validator network at 8,000 SAC payments per second: 29 us per load with the cache against 14 us without). RandomEvictionCache gets peek(), a const lookup that records nothing, and LiveBucketIndex::getCachedEntry uses it under the shared lock. Inserting an entry still takes the lock exclusively. A hit no longer makes its entry younger for eviction: when a cache is smaller than its bucket's accounts, eviction goes by insertion time. A cache sized to hold all accounts (the case this is for) never evicts.
…lot and executes its prepared set before the trigger The first-round leader's build of a transaction set sat between the close of the previous ledger and its proposal: a set names its parent's header hash, so the build could not start before that ledger had closed. 1. Selection during the previous ballot. The node that expects to lead round 1 of ledger N+1 selects its set for N+1 while N is still in its ballot: the pre-validation job (which validates candidates on the state the node's speculative execution of N left) goes on, on its last task's thread, to trim and surge-price what it pulled and to write the set's XDR without a parent. When N has closed with the executed value and a build would have to start at once (the trigger is due, or less than a build's lead time away), the node only names N as the parent, hashes the set, hands it to the overlay and pushes it (HerderImpl::prepareEarlyTxSet). The selection is used only if N closed with exactly the executed value, the limits it assumed are those of the closed ledger, its time-dependent transactions are valid for the close time the set gets, and a build of its size would take time (an empty selection is never proposed). Otherwise the ledger gets one build as before. 2. The leader executes its own set as soon as it is prepared. The value the trigger will propose is known when the set is (HerderImpl:: executePreparedTxSet), so the ledger goes to the apply thread then, not when the first ballot statement is emitted. If that execution finds the set invalid before the proposal, the set is dropped. 3. The builder's check of the written set (the XDR round trip and the validation of the re-parsed set) runs on a background thread for sets the herder builds as a leader on protocols that validate tx sets at apply (TxSetSelfCheck). The structure of the selection and the shape of the XDR are still checked before the set leaves the builder. makeTxSetFromTransactions is split for this: selection takes the limits (TxSetBuildLimits) and the validation context explicitly instead of reading the LCL through the application, so that it can run for a ledger that is not the LCL and off the main thread. Two faults of the base are fixed on the way: the pre-validation's transaction frames referred to a network ID owned by a task's closure, and a build kept one transaction per source account within each phase but not across the two phases (the builder's own check then threw). No change to apply, commit, tx-set validation or the wire format.
…d the deferred self-check New test cases (tag [early-txset]) in ProposalPrevalidationTests.cpp. Two leader-build tests in HerderTests.cpp count builds instead of mempool pulls (the leader's early execution pulls candidates for the ledger after it), and "SCP Driver / validateValue txSet cached" no longer depends on which of its two validators leads (it failed on the base at --base-instance 1250).
…e validated at apply
…fore it closes, and pulls less for a build Two changes to how a leader gets its transaction set, on top of the reused parts of the closed attempt (builder split, deferred check of the written set, early execution of a prepared set). 1. A smaller build. A build pulls the capacity of a set plus a margin (EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT, default 10) from the mempool instead of twice the capacity. If a phase loses candidates (invalid, or a second transaction of an account) and ends below its capacity while the mempool has more, the build pulls again, twice as much, at most three times. The structural check of the selection runs on a second thread beside the XDR write and is awaited before the set is handed to anyone. The re-read of the written set is left to the node's own execution of its value where it executes its votes, and runs in the background otherwise. 2. Early selection (EXPERIMENTAL_EARLY_TX_SET_SELECTION, default on). When a node votes for a value for ledger N and expects to lead round 1 of N+1, it plans a background job that selects the set for N+1 while N is in its ballot: on the last closed ledger (N-1) and a prediction of what N's set does to it (sequence numbers consumed, full fees charged). The job starts as late as it can and still finish before the set is needed. The prediction is never trusted. At the build for N+1, if N closed with exactly the value the job was for, the selection is completed: every one of its transactions is validated against N as it closed, with the set's close time, by the function that validates a received set; what fails is removed; while a phase has room, further candidates (what the job did not select, then what the mempool holds that the job did not see) are validated the same way and added, to a parallel Soroban phase one by one (ParallelPhaseExtender), to a sequential phase by selecting it again; base fees are set by the rule of a build; and the set goes through the same checks and hand-off as a built one. Anything else (N closed with another value, with upgrades, the job not finished, a failed check) is a build. Tests: herder/test/EarlyTxSetTests.cpp. A completed set is validated in full on the closed ledger, read back from its bytes, for: the candidates being exactly N's transactions; the same accounts with their next sequence numbers; random overlaps; a transaction of N failing; an account drained, merged away or re-keyed by N; fee bumps; N's set skipped at apply; N closing with another value, close time or an upgrade; a job that is late, still running, or overlapping the next one; time bounds. Three of them show the set invalid when the validity pass is switched off. Fullness: top-up with late arrivals, refill after removals and for builds with 0 to 90% invalid candidates, Soroban top-up and conflicts, base fees after removal and refill. Nothing in apply, commit, set validation or the wire format changes.
…ne that lost candidates looks deeper whatever the mempool's top holds of its kind The stop rules of the completion's top-up and of a build's refill had two shortcuts that could leave a phase short while the mempool held valid transactions for it: - eight or more valid left-overs that could not be added closed the phase, also when they only had nowhere to go (a full cluster of one entry, a lane of its own) and later arrivals would have fit; - a phase that lost candidates pulled deeper only if the mempool's top held at least the phase's capacity of its kind (compared in transactions against operations), which a mempool top of the other phase prevents. Now, as in a build: a phase stops at its capacity; after one look at the mempool's top if it lost nothing (lost: removed by the validity pass, invalid, or of an account already in the set; a valid candidate that did not fit is not a loss); and otherwise when a deeper pull holds nothing new of its kind. Tests: many left-overs with nowhere to go; a completion and a build with a mempool top of the other phase.
Two lines of the load generator commits that clang-format 20 (CI's version) breaks differently. No other change.
…ixth experiment (the next tx set selected before the close) - What this branch is: experiment 6 (a smaller build; the leader of ledger N+1 selects its set from ledger N's set during N's ballot and completes it on the closed ledger), with its two supporting changes. - Results: the bucket list settings on 6 validators; the base with the meta off at 90 validators (801 ms per ledger at 6,000 TPS, 873 ms at 8,000 TPS); experiment 6 at 8,000 TPS (build 176 -> 121 ms, 858 ms per ledger, no invalid or short set), and why the shorter build does not show in the block time. - How it was measured, limits of the numbers, known limitations, test status.
…rom the schedule's nodes and the slot's seed EXPERIMENTAL_TX_SET_TREE = off | star | hash | rtt (default off). Until now the round's leader writes its whole (compressed) tx set to every peer: 89 copies at 90 validators. With a tree it writes it to its children only, and every node passes the bytes on to its own children. - The tree (overlay/src/flood/txset_tree.rs). For a slot and a leader: the leader is the root, the other nodes of the leader schedule follow in the order of SHA256(seed || leader || node), and position i's children are the positions F*i+1 .. F*i+F (the complete F-ary tree, the least depth a fanout of F allows). The seed is the seed of the slot's leader schedule. F is EXPERIMENTAL_TX_SET_TREE_FANOUT; 0 = the smallest F that reaches every node in two hops (9 for 90 nodes). Every node derives the same tree from what it already agrees on. - Core tells its overlay the nodes and the seeds of the next two slots at every ledger close (SetTxSetTree, HerderImpl::updateTxSetTree). - A pushed frame carries a header in front of the ordinary tx-set frame: kind, slot, leader, hop count and the leader's send time (and, in rtt mode, the order of the nodes). - A node relays a tagged frame to its children before it decompresses or checks it, if it knows the tree of that slot, the sender is its parent in it (by the validator identity of the connection) and it has not relayed that frame. It then verifies and files the set by the hash of its content as before. - A node that expects a set through the tree waits 300 ms from the first mention of the set's hash before it asks a peer for it; otherwise every node the leader did not push to would ask it when the proposal arrives. - star: to every peer as before, with the header (the baseline for measurements). A leader that cannot reach one of its children, or does not know the slot's tree, pushes a star. - rtt: the leader measures the round trip to every peer (a 13-byte probe on the tx-set stream), groups its peers by the gaps in the sorted times, gives every group relays in proportion to its size and fills a relay's children from its own group. The order travels in the header. - Metrics overlay.txset-tree.* (copies written as root and as relay, received, refused, duplicates, unreachable children, requests put off and those sent after the wait, delay since the leader's send in total, per hop count and in buckets) and overlay.txset-push.sent-bytes. Nothing in SCP, the schedule, apply or a tx set's content changes: a set is still identified by the hash of its content, computed by the receiver, and reaches Core only when an SCP statement names that hash. Tests: the tree's logic (order, least depth, header, relay decision, wait, rtt order), seven overlay nodes on loopback (delivery down the tree, a relay that does not relay, a frame from a node that is not the parent, star, an explicit order, probes), and 7-validator networks under consensus in each mode.
…odes below a busy child After an independent review of the first commit: - Only a node that this node's Core expects to lead the slot can root a tree: Core sends the leaders of the first three rounds of the next slot, and for the slot after it those plus the first three of a new order (SetTxSetTree carries them per slot). A frame that names another leader is filed and not relayed. - An order chosen by the sender (rtt mode) is followed only by nodes in rtt mode, only for a tree of two levels, and only if it refers to the same list of nodes (a digest of the list travels with the order). - A node relays at most two different frames per slot and leader (it told copies apart by their first and last bytes before, which a sender can vary), and has at most one relay in flight per child. - A frame whose body decode would refuse for its size is not relayed. - A probe answer is a measurement only if it answers the probe this node sent that validator last; one answer to a peer's probes is in flight at a time. - If the leader cannot send a child the set (an earlier push to it is still in flight), the nodes below that child get it from the leader directly, as a star. Before, they waited for a relay that had nothing to relay. - A node that is not one of the tree's nodes does not wait before it asks. - The layout by round-trip times is computed outside the tree's lock. - Counters: overlay.txset-tree.child-busy, relay-busy, relay-limited. The delay-le-* buckets are disjoint ranges; their descriptions say so. Tests for each of these (tree module, seven overlay nodes on loopback, the IPC payload's layout).
…elay in the middle The leader grouped its peers by gaps in the sorted round-trip times, where a gap was 8 ms or a fifth of the larger time, and gave a group's relays the nearest nodes of the group. Two regions at similar distances from the leader (us-west-2 and ap-southeast-1 seen from eu-central-1, for one) became one group whose relays were all in the nearer region: the nodes of the other one got the set over two long hops. Now a group ends at a gap of more than 8 ms only, and a group's nodes, sorted by round-trip time, are cut into consecutive runs, one per relay, with the run's middle node as its relay. Regions at the same distance still differ by a little, so the runs mostly fall apart along them. With more groups than relays all nodes are one group, laid out the same way (before: nearest first, which is not a two-level tree a receiver follows). In a model with typical round trips between four AWS regions (the item's results/dissemination-model.py) the mean delay of a set goes from 95 to 84 ms and the 90th percentile from 178 to 134 ms, which is what a leader could do if it knew every distance.
The test sent three frames back to back and expected two of them to reach a child; with one relay in flight per child the second could be skipped.
…re-coded pieces EXPERIMENTAL_TX_SET_TREE="coded": the leader cuts the frame it would push into k pieces, adds m-k Reed-Solomon parity pieces (16 pieces of which any 8 rebuild the frame at 90 validators), signs the root of their Merkle tree with its overlay key and writes every piece to that piece's relays; a relay passes its piece on. A node gets its m pieces from m different senders, so up to m-k of them can be down, slow or hostile without the node having to wait and ask. A piece is kept and passed on only if it comes from the node that owes it in the layout, names a leader Core expects for the slot, carries the leader's signatures and a Merkle path to the signed root. A rebuilt frame goes the way of every received frame (decompress, strict parse, content hash). EXPERIMENTAL_TX_SET_TREE_WITHHOLDERS_FOR_TESTING=N makes the N validators with the smallest node IDs pass nothing on, in the tree modes and in coded mode (tests and fault runs).
…'s seed, and do not code as a leader nobody expects From the review of the coded mode: - A piece frame is counted against its sender (six per slot and leader) before the layout is consulted, whatever the frame turns out to be, and the layout is worked out once per slot and leader. - The signed message holds the seed of the slot's layout, so a root signed in another history of a network with the same name and keys is refused. - A node that Core does not name as a leader of the slot pushes its set to everyone instead of cutting pieces (or rooting a tree) that every node would refuse. - A rebuilt set counts as received from its leader, not from the sender of the piece that completed it. - EXPERIMENTAL_TX_SET_TREE_WITHHOLDERS_FOR_TESTING is a testing-only option.
… signed its root From the second review of the coded mode: - The overlay key that signed a root is kept with the root, and a set rebuilt from pieces counts as received from that overlay, connected or not. Before, a node not connected to the leader charged the sender of the piece that completed the set, which the leader could choose. - A root gets its place only from a frame whose signatures were checked. - The leader named in a piece must be one of the layout's nodes before the frame is counted. - A node that is not a named leader of the slot logs that it pushes to every peer (TXSET_CODED_STAR, TXSET_TREE_STAR). - Tests: the unexpected leader in hash mode, the attribution, a settle before a counter is read. Docs and comments that still described the old behaviour of an unnamed leader.
…eal apply at 8,000 and 10,000 TPS, against the spanning tree and the push, with validators withholding, and what limits 10,000 TPS
…ne config line put back as clang-format has it
1024 16-bit elements per member, from a ChaCha20 keystream keyed by the SHA-256 of a domain tag and the member; members are added and removed element-wise, so the hash of a set does not depend on the order of its members or on how the set is split into parts.
EXPERIMENTAL_SHARDED_TOP_BUCKET = N (0, the default, is off). The hash of a level-0 bucket becomes the LtHash16 of its records instead of the SHA-256 of its file, so a ledger's entries can be hashed and written in parts: they are cut into shards by key (N for contract data and code, N for TTL entries, N for everything else), and every shard is sorted, written and synced, merged with the shard of the ledger before, hashed and indexed on its own thread. The bucket's file, byte for byte the file level 0 has without the setting, is written afterwards by a thread of its own; whoever needs it waits for it. It is what spills into level 1 and what is published, so levels 1 and deeper keep SHA-256 and their buckets. The setting changes the bucket list hash of the ledger header: a network has it on at every node or at none. The number of shards is each node's choice. A node that stops between a ledger's commit and the end of the bucket's file rebuilds the file at startup from the ledger's shard files and the bucket they were added to. Catchup takes a bucket named by either hash when the setting is on. STELLAR_TEST_SHARDED_TOP_BUCKET=N runs the tests that take their config from getTestConfig with the setting.
… review fixes - A bucket file is taken under the commutative hash only if it is the canonical file of its records: the metadata entry first, entries in the strict order of their keys, canonical XDR, nothing after the last record. The hash alone does not see the order, and catchup would have adopted a file that the next merge aborts on. - The file's writer uses nothing of the bucket manager, and the bucket manager waits for it before it removes its directories. - The shards' worker pool is not pinned (BatchExecutor(pinWorkers = false)): its batches can overlap those of the application's executor. It takes BATCH_EXECUTOR_TASKS. - The setting is refused on the public network, and a level 0 written without it is reported at startup instead of in a ledger close. - The shard of a key is mixed with a value only the process knows. - The digest of an LtHash16 state has its own tag. - A bucket without a metadata entry has version 0, as its file says. - bucket.top-shard.gc-wait times the wait of the bucket garbage collection. Tests: the canonical file check; the equivalence test also at the last protocol without and the first with INIT and metadata entries.
…of the state update - The canonical file check reads the file record by record instead of whole: with the setting on, catchup hands it every live bucket that is not named by its SHA-256, of any size. - A file that is not canonical is logged by catchup like any other bucket that fails verification. - The bucket manager, before it removes its directories, waits for the file of every bucket made ahead of its file, not only for level 0's current one (a bucket list may have been assumed over it). - bucket.top-shard.gc-wait is registered only with the setting. Tests: two more non-canonical files; the setting over state written without it is refused at startup. A hidden benchmark of the in-memory Soroban state update, which the ledger commit runs beside the bucket list add and waits for.
… (experiment 8): real apply at 10,000 TPS with the setting on and off, and what limits it after
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prototype, draft, config-gated and off by default. Level 0 of the live bucket list is hashed with a commutative hash and written as parallel shards. On 90 validators at 10,000 TPS with real apply the level-0 add goes from 97.6 ms to 21.1 ms per ledger (-78%) and the commit from 203 ms to 162 ms (-21%). The network does not get faster for it: a full ledger every 1,003.5 ms instead of 1,009 ms, verdict FAIL on both sides. The text below says why, and what limits 10,000 TPS after this.
On
overlay-v2-shared, stacked on #5506 (open draft,371474595): the diff also shows its commits and #5505's.What this diff holds beyond #5506 (5 commits):
129540736:LtHash16(src/crypto/LtHash.{h,cpp}).1a9348463: the sharded level 0 (src/bucket/ShardedTopBucket.{h,cpp}), buckets that exist ahead of their file, the restart rebuild, catchup, the setting, tests, a hidden benchmark.be560d831: fixes from the first review. Both network runs below were made on this commit.6f58498f6: fixes from the review of that commit, off the measured path (how catchup reads a file it checks, what the bucket manager waits for when it is destroyed), tests, a hidden benchmark of the in-memory state update. Not on a network.1a69a9d51: README (experiment 8).What it does
Today a ledger's entries become one level-0 bucket that one thread sorts, serializes, writes and hashes with SHA-256 inside the ledger's commit. At 10,000 payments per ledger that is 97 ms of a 203 ms commit.
With
EXPERIMENTAL_SHARDED_TOP_BUCKET = N(0, the default, is off):Two things asked for that this does differently:
Results
Fixture: experiment 7's run at 10,000 TPS. 90 validators (m6id.8xlarge) in four regions, 800 ms target, 10,000 native-asset contract payments per second executed for real, meta off, 16 clusters, the tx set as coded pieces, no faulty nodes. One package on both sides (
be560d831,30.0.1-3698.be560d8314.noble~perf.sharded.top.bucket.writes~perftests); the only difference is the setting (8 shards per group, or 0). One run each, 22 minutes apart. Timers are means per ledger pooled over the 90 nodes between 150 s and 315 s of the load, where ledgers are full.bucket.batch.addtime)ledger.apply.commit)ledger.apply.execute)ledger.ledger.close)Every check other than the windows passed in both runs: inclusion on every node, in sync, no fork, error counters, no restarts.
What I predicted before writing code, and what came out:
Why the commit gains 42 ms and not 76. The commit waits for two things that run side by side: the add, and the update of the in-memory Soroban state with the same entries. With the add at 21 ms the state update is what the step waits for. It has no timer; on a development machine it takes 31 ms per ledger for this load, on one thread (the hidden benchmark of
6f58498f6). That it is the bound on the network is an inference from these two numbers.Why the time between full ledgers gains 5 ms.
What the setting buys shows before the network saturates. For the first 125 ledgers (about 105 s) the network with the setting keeps up with the load at 843 ms a ledger; without it, it is at 939 ms from the start. Both drift up, because the close after consensus grows during the load (216 to 459 ms and 267 to 436 ms over 300 ledgers on one node), until the tx set cap is reached. One run each: the direction is the same in all five groups of 25 ledgers, the size is one observation.
Where 10,000 TPS is limited after this (with the setting, per full ledger):
Local benchmark (a development machine, 16 cores, 30,000 entries per ledger shaped like the load, files synced, hidden tests tagged
[bucketbench]): the add 71.7 ms without and 14.4 ms with 8 shards per group (-80%). It takes about 130 ms of thread time on 15 threads, a third of it the hash (1.0 us per record). 4 shards per group: 20.9 ms; 16: 13.9 ms; 32: 14.4 ms.Correctness
6f58498f6(a timer that was registered either way).mergeInternal, the code of every merge; no key is in two shards, so the shards interleaved by key are what one merge of the whole gives.Tests
src/crypto/test/LtHashTests.cpp(4 cases: order and split independence, add and remove, a pinned digest) andsrc/bucket/test/ShardedTopBucketTests.cpp(5 cases: the same buckets under another hash; a canonical file only; the setting over state written without it; the restart rebuild; four validators with different shard counts).--all-versions: pass atbe560d831and at the head (setting off).STELLAR_TEST_SHARDED_TOP_BUCKET=3): the first such run found two real bugs (a metrics timer freed under the file's writer; the startup check not waiting for the file), fixed before the first commit. At the head 4 batches fail, none a defect of the change: one pinned header hash, two bucket index tests that pin level 0's old representation, and one test of Prototype: the leader's tx set travels as erasure-coded pieces, or down a spanning tree (real-apply results, README) #5506 that requires a particular node to be a relay at least once in 28 ledgers (the trees are seeded by ledger hashes; without the setting it passes with exactly one such ledger).clang-format20 on the changed C++ files. The 12 changed non-test objects compile withoutBUILD_TESTS.build-linuxjobs stop in the format check on files of the base that this PR does not touch.Review
Two independent reviews. The first, of
1a9348463: no way found for honest nodes to disagree on a hash or a bucket, the crash invariant holding as coded, no deadlock; one critical finding, catchup took any file whose records have the commutative hash, also with the records in another order, which the next merge aborts on (fixed inbe560d831with the canonical-file check), and a shutdown race of the file's writer (fixed). The second, ofbe560d831: no blocker; the canonical check read whole files into memory, and the wait at shutdown covered only level 0's current bucket (both fixed in6f58498f6). Left open on purpose:Known limits
Checklist
overlay-v2-sharedatd9ae5fcecplus Prototype: the leader selects its next tx set from the current one before it closes, and pulls less for a build (real-apply results, README) #5505 and Prototype: the leader's tx set travels as erasure-coded pieces, or down a spanning tree (real-apply results, README) #5506; no merge commits)clang-format(20.1.8) on the changed files