Repository navigation
Conversation
This was referenced Oct 6, 2026
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
from
October 8, 2026 08:18
004654c to
1641df7
Compare
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
3 times, most recently
from
October 9, 2026 06:02
7264ea8 to
0588eeb
Compare
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
from
October 15, 2026 19:15
0588eeb to
94c4c9a
Compare
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
from
October 19, 2026 13:00
94c4c9a to
6d34f51
Compare
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
2 times, most recently
from
October 21, 2026 09:12
adeffd5 to
f892112
Compare
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
2 times, most recently
from
October 21, 2026 14:04
99657ab to
92d2b2d
Compare
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
from
October 21, 2026 14:07
92d2b2d to
67060a6
Compare
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
from
October 28, 2026 08:05
b0fc425 to
23539e1
Compare
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
from
November 4, 2026 10:07
23539e1 to
737aadc
Compare
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
3 times, most recently
from
November 6, 2026 06:34
bbec3ff to
ab7ce41
Compare
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
from
November 12, 2026 09:00
160dcd3 to
6d4aa22
Compare
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
3 times, most recently
from
November 12, 2026 13:32
3004fe2 to
c5a38af
Compare
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
from
November 12, 2026 14:38
c5a38af to
9fae621
Compare
tamirms
force-pushed
the
tamirms/hot-tier-window-index
branch
from
November 13, 2026 07:25
9fae621 to
843cf9c
Compare
…he hot tier
The tx-hash index of the window still being filled is rewritten at every
chunk boundary until the window terminates, so it is the one cold artifact
that is not write-once: on a throughput-capped volume the rewrite competes
with every cold read for as long as it runs. It now lives with the hot
tier: a coverage that is not terminal is written under the hot root, in a
txhash-index-{window} directory beside the chunk DBs, and the terminal
coverage is a cold artifact under the txhash index root, decided from the
coverage alone in Layout.TxHashIndexDir. The build, the sweep and the
reader all resolve the path from the coverage, so the terminal build lands
in the cold root and the same commit batch demotes the last hot-tier
coverage for the sweep. The layout now owns the window arithmetic, which
the catalog takes from it, so the two cannot disagree.
The events index build's spilled runs move from the index root to a
scratch directory under the hot root, so the cold roots hold only
artifacts and moving the hot root moves every build's transient writes
with it. The sample config names what belongs on the local disk (hot,
txhash_raw, captive core) and that the catalog stays on the durable
volume.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The gettransaction design and the explorer still placed the open window's index under txhash/index/, where only a complete window's index lives now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9U6S8NJST6PCjAadTD4zu
Every chunk boundary rereads the window's .bin files in full. On the volume that serves cold queries, those reads make query reads wait: on a baseline gp3 volume, the rebuild's read pattern raised the p50 of a 4 KiB read from 0.6 ms to 5.9 ms, and left it at 0.6 ms on a second volume. On the local NVMe disk, a node replacement would mean regenerating them from the ledger packs, which takes hours per window. So the advice is now a durable volume of their own, sized for a first start, which holds every window's .bin files at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9U6S8NJST6PCjAadTD4zu
…node The sample config suggested putting events_index on fast disk without saying what losing that disk costs. The catalog tracks a chunk's events pack and its index as one artifact, so a lost index means rebuilding every chunk's events from the ledger packs: for pubnet's 4.9 TiB of packs, about 11 hours of reads at the gp3 baseline. The advice now keeps the speedup but only for a disk that survives a node replacement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9U6S8NJST6PCjAadTD4zu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
rpc v2 is meant to run with its data split across storage tiers: a small local NVMe disk for what is recent or rewritten, and durable volumes (EBS today, possibly object storage later) for everything else. The
[storage]section already lets each store sit on its own disk. What makes the split work is that the cold stores are write-once: a chunk's ledger pack, events pack, events index and terminal tx-hash index are each written once, at freeze, and only read after that. The slow volume sees its reads plus one sequential write per chunk, and its throughput goes to serving queries.Two things did not fit that model.
The tx-hash index of the window still being filled. A window is 1000 chunks, and its index is rebuilt from scratch at every chunk boundary until the window is complete: the rebuild reads every
.bininput of the window and writes a new.idxthat grows with the window, then the previous.idxis swept. That is a sequential rewrite landing on the cold volume at every boundary, for the life of every window. On EBS the IOPS and bandwidth quotas are shared between reads and writes, so for as long as the rewrite runs, getTransaction and getEvents cold reads queue behind it. The gp3 baseline is 125 MiB/s and 3,000 IOPS, and a sequential rewrite of a growing index takes the bandwidth quota for as long as it runs. This index is also the first one every cold getTransaction probes, since lookups try the newest window first, so on EBS every cold lookup would start with up to three dependent EBS reads.The events index build's spilled runs. Since #1098 the build keeps at most two slabs of postings in memory and spills the rest as sorted runs on disk, which it merges once at finalize into a terms file it then reads twice. The runs and the terms file were written beside the index files, so every build, including the parallel backfill builds, put its transient I/O on the cold volume.
What changes
Both now live with the hot tier, which is where everything else that is rewritten already lives.
Layout.TxHashIndexDirtakes the coverage, not the window id. A coverage that is not terminal is written under the hot root, intxhash-index-{window}/beside the chunk DBs. A terminal coverage is a cold artifact undertxhash_index/{window}/. The build, the sweep and the reader all resolve the path from the coverage, so the terminal build lands in the cold root and the same commit batch demotes the last hot-tier coverage, which the sweep then removes from under hot. No catalog key or value changes: whether a coverage is terminal was already derived from the key.Layoutowns the tx-hash window arithmetic (WithTxHashIndex,TxHashIndex).catalog.Openreads it from the layout instead of taking a second parameter, so the two cannot disagree.{hot}/scratch/{chunk}-index.runs/instead of the index root. The cold roots hold only artifacts, and movinghotmoves every build's transient writes with it; there is no second key to forget.txhash_raw, and the sample config advises a fast disk forevents_indexonly if that disk survives a node replacement (both below). The design docs show the index being rebuilt under the hot root.A deployment on NVMe and EBS
The rule behind the split is write pattern, not age.
hottakes every ledger's writes (the chunk DBs), the per-boundary rewrite of the window index, and the builds' scratch. Captive core churns its own database. Everything on the NVMe disk can be rebuilt: the chunk DBs from the ledger backend, the window index from the.binfiles. The catalog is the one thing nothing can rebuild from the files, so it stays on the durable volume with the artifacts it describes. The remaining stores default underdefault_data_dir,events_indexincluded. Its indexes are small and probed at random, so a fast disk speeds up wide filtered event queries, but on the NVMe disk a node replacement would mean rebuilding every chunk's events from the ledger packs: for pubnet, about 11 hours of reads at the gp3 baseline.txhash_rawgets a durable volume of its own because of how it is read. Its.binfiles are written once per chunk, then read in full at every boundary until the window is complete.Measured on an m6id.2xlarge with gp3 volumes at the baseline:
.binfiles: 16 goroutines of direct 76 KiB reads, about 121 MiB/s. It is not a full build.On a volume of its own, that volume's throughput setting is what paces the rebuild: today's pubnet window, about 90 GB of
.binfiles, takes about 11 minutes at the baseline, by arithmetic. A first start holds every window's.binfiles at once, about 240 GB for pubnet's history today by an estimate from the index sizes, so size the volume for that rather than for one window.After a node replacement
A node replacement wipes the NVMe disk, and with this change the open window's index goes with the hot data. The catalog still lists that index as frozen, so nothing rebuilds it, and cold lookups would fail until it exists again. Startup already fails today when a hot chunk's DB is missing. The planned startup check for a wiped hot root has to demote this index's key too, so that the startup backfill rebuilds it before reads open. Whichever of the two changes lands second adds that.
🤖 Generated with Claude Code