Skip to content

storage: put the tx-hash index being rebuilt and build scratch under the hot root - #1099

Draft
tamirms wants to merge 4 commits into
tamirms/events-cold-index-runsfrom
tamirms/hot-tier-window-index
Draft

tamirms wants to merge 4 commits into
tamirms/events-cold-index-runsfrom
tamirms/hot-tier-window-index

Conversation

@tamirms

@tamirms tamirms commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

rpc v2 is meant to run with its data split across storage tiers: a small local NVMe disk for what is recent or rewritten, and durable volumes (EBS today, possibly object storage later) for everything else. The [storage] section already lets each store sit on its own disk. What makes the split work is that the cold stores are write-once: a chunk's ledger pack, events pack, events index and terminal tx-hash index are each written once, at freeze, and only read after that. The slow volume sees its reads plus one sequential write per chunk, and its throughput goes to serving queries.

Two things did not fit that model.

The tx-hash index of the window still being filled. A window is 1000 chunks, and its index is rebuilt from scratch at every chunk boundary until the window is complete: the rebuild reads every .bin input of the window and writes a new .idx that grows with the window, then the previous .idx is swept. That is a sequential rewrite landing on the cold volume at every boundary, for the life of every window. On EBS the IOPS and bandwidth quotas are shared between reads and writes, so for as long as the rewrite runs, getTransaction and getEvents cold reads queue behind it. The gp3 baseline is 125 MiB/s and 3,000 IOPS, and a sequential rewrite of a growing index takes the bandwidth quota for as long as it runs. This index is also the first one every cold getTransaction probes, since lookups try the newest window first, so on EBS every cold lookup would start with up to three dependent EBS reads.

The events index build's spilled runs. Since #1098 the build keeps at most two slabs of postings in memory and spills the rest as sorted runs on disk, which it merges once at finalize into a terms file it then reads twice. The runs and the terms file were written beside the index files, so every build, including the parallel backfill builds, put its transient I/O on the cold volume.

What changes

Both now live with the hot tier, which is where everything else that is rewritten already lives.

  • Layout.TxHashIndexDir takes the coverage, not the window id. A coverage that is not terminal is written under the hot root, in txhash-index-{window}/ beside the chunk DBs. A terminal coverage is a cold artifact under txhash_index/{window}/. The build, the sweep and the reader all resolve the path from the coverage, so the terminal build lands in the cold root and the same commit batch demotes the last hot-tier coverage, which the sweep then removes from under hot. No catalog key or value changes: whether a coverage is terminal was already derived from the key.
  • Layout owns the tx-hash window arithmetic (WithTxHashIndex, TxHashIndex). catalog.Open reads it from the layout instead of taking a second parameter, so the two cannot disagree.
  • The events index build spills its runs, and merges them, under {hot}/scratch/{chunk}-index.runs/ instead of the index root. The cold roots hold only artifacts, and moving hot moves every build's transient writes with it; there is no second key to forget.
  • The sample config and the gettransaction design now advise a durable volume of its own for txhash_raw, and the sample config advises a fast disk for events_index only if that disk survives a node replacement (both below). The design docs show the index being rebuilt under the hot root.

A deployment on NVMe and EBS

[storage]
# The durable volume holds the catalog and every write-once artifact:
# ledger packs, events packs and indexes, terminal tx-hash indexes.
default_data_dir = "/mnt/ebs/rpc-v2"
# A second, smaller durable volume holds the .bin files every rebuild reads.
txhash_raw = "/mnt/ebs-txhash/rpc-v2/txhash-raw"
# The local NVMe disk holds what is rewritten or transient.
hot = "/mnt/nvme/rpc-v2/hot"

[ingestion]
captive_core_storage_path = "/mnt/nvme/rpc-v2/captive-core"

The rule behind the split is write pattern, not age. hot takes every ledger's writes (the chunk DBs), the per-boundary rewrite of the window index, and the builds' scratch. Captive core churns its own database. Everything on the NVMe disk can be rebuilt: the chunk DBs from the ledger backend, the window index from the .bin files. The catalog is the one thing nothing can rebuild from the files, so it stays on the durable volume with the artifacts it describes. The remaining stores default under default_data_dir, events_index included. Its indexes are small and probed at random, so a fast disk speeds up wide filtered event queries, but on the NVMe disk a node replacement would mean rebuilding every chunk's events from the ledger packs: for pubnet, about 11 hours of reads at the gp3 baseline.

txhash_raw gets a durable volume of its own because of how it is read. Its .bin files are written once per chunk, then read in full at every boundary until the window is complete.

  • On the NVMe disk, a node replacement would mean regenerating them from the ledger packs before reads open: 13.5 s per chunk at the gp3 baseline, so hours for a full window.
  • On the volume that serves queries, the rebuild reads at the volume's limit, and every query read waits behind its reads:
Rebuild reads Query read p50 Query read p99
None 0.6 ms 1.0 to 1.5 ms
On the same volume 5.9 ms 11.3 ms
On a second volume 0.6 ms 0.9 ms

Measured on an m6id.2xlarge with gp3 volumes at the baseline:

  • The rebuild side is the build's read pattern over 535 real pubnet .bin files: 16 goroutines of direct 76 KiB reads, about 121 MiB/s. It is not a full build.
  • A query read is one direct 4 KiB read at a random offset, 100 per second, for 30 seconds per row.

On a volume of its own, that volume's throughput setting is what paces the rebuild: today's pubnet window, about 90 GB of .bin files, takes about 11 minutes at the baseline, by arithmetic. A first start holds every window's .bin files at once, about 240 GB for pubnet's history today by an estimate from the index sizes, so size the volume for that rather than for one window.

After a node replacement

A node replacement wipes the NVMe disk, and with this change the open window's index goes with the hot data. The catalog still lists that index as frozen, so nothing rebuilds it, and cold lookups would fail until it exists again. Startup already fails today when a hot chunk's DB is missing. The planned startup check for a wiped hot root has to demote this index's key too, so that the startup backfill rebuilds it before reads open. Whichever of the two changes lands second adds that.

🤖 Generated with Claude Code

@tamirms
tamirms added this pull request to stack #1089 October 6, 2026 09:55
@tamirms tamirms changed the title storage: keep the window index being rebuilt and build scratch with the hot tier storage: put the tx-hash index being rebuilt and build scratch under the hot root Oct 6, 2026
@tamirms
tamirms removed this pull request from stack #1089 October 8, 2026 08:18
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch from 004654c to 1641df7 Compare October 8, 2026 08:18
@tamirms
tamirms added this pull request to stack #1102 October 8, 2026 08:19
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch 3 times, most recently from 7264ea8 to 0588eeb Compare October 9, 2026 06:02
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch from 0588eeb to 94c4c9a Compare October 15, 2026 19:15
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch from 94c4c9a to 6d34f51 Compare October 19, 2026 13:00
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch 2 times, most recently from adeffd5 to f892112 Compare October 21, 2026 09:12
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch 2 times, most recently from 99657ab to 92d2b2d Compare October 21, 2026 14:04
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch from 92d2b2d to 67060a6 Compare October 21, 2026 14:07
@tamirms
tamirms removed this pull request from stack #1102 October 27, 2026 10:19
@tamirms
tamirms added this pull request to stack #1157 October 27, 2026 10:21
@tamirms tamirms moved this from To Do to In Progress in Platform Scrum Oct 27, 2026
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch from b0fc425 to 23539e1 Compare October 28, 2026 08:05
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch from 23539e1 to 737aadc Compare November 4, 2026 10:07
@tamirms
tamirms removed this pull request from stack #1157 November 4, 2026 10:20
@tamirms
tamirms added this pull request to stack #1184 November 4, 2026 10:21
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch 3 times, most recently from bbec3ff to ab7ce41 Compare November 6, 2026 06:34
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch from 160dcd3 to 6d4aa22 Compare November 12, 2026 09:00
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch 3 times, most recently from 3004fe2 to c5a38af Compare November 12, 2026 13:32
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch from c5a38af to 9fae621 Compare November 12, 2026 14:38
@tamirms
tamirms force-pushed the tamirms/hot-tier-window-index branch from 9fae621 to 843cf9c Compare November 13, 2026 07:25
tamirms and others added 4 commits November 16, 2026 15:35
…he hot tier

The tx-hash index of the window still being filled is rewritten at every
chunk boundary until the window terminates, so it is the one cold artifact
that is not write-once: on a throughput-capped volume the rewrite competes
with every cold read for as long as it runs. It now lives with the hot
tier: a coverage that is not terminal is written under the hot root, in a
txhash-index-{window} directory beside the chunk DBs, and the terminal
coverage is a cold artifact under the txhash index root, decided from the
coverage alone in Layout.TxHashIndexDir. The build, the sweep and the
reader all resolve the path from the coverage, so the terminal build lands
in the cold root and the same commit batch demotes the last hot-tier
coverage for the sweep. The layout now owns the window arithmetic, which
the catalog takes from it, so the two cannot disagree.

The events index build's spilled runs move from the index root to a
scratch directory under the hot root, so the cold roots hold only
artifacts and moving the hot root moves every build's transient writes
with it. The sample config names what belongs on the local disk (hot,
txhash_raw, captive core) and that the catalog stays on the durable
volume.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The gettransaction design and the explorer still placed the open window's
index under txhash/index/, where only a complete window's index lives now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6S8NJST6PCjAadTD4zu
Every chunk boundary rereads the window's .bin files in full. On the
volume that serves cold queries, those reads make query reads wait: on a
baseline gp3 volume, the rebuild's read pattern raised the p50 of a 4 KiB
read from 0.6 ms to 5.9 ms, and left it at 0.6 ms on a second volume. On
the local NVMe disk, a node replacement would mean regenerating them from
the ledger packs, which takes hours per window. So the advice is now a
durable volume of their own, sized for a first start, which holds every
window's .bin files at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6S8NJST6PCjAadTD4zu
…node

The sample config suggested putting events_index on fast disk without
saying what losing that disk costs. The catalog tracks a chunk's events
pack and its index as one artifact, so a lost index means rebuilding every
chunk's events from the ledger packs: for pubnet's 4.9 TiB of packs, about
11 hours of reads at the gp3 baseline. The advice now keeps the speedup
but only for a disk that survives a node replacement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6S8NJST6PCjAadTD4zu
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant