Skip to content

rocksdb: load a file of sorted entries into a column family - #1096

Open
tamirms wants to merge 1 commit into
mainfrom
tamirms/rocksdb-bulk-load
Open

tamirms wants to merge 1 commit into
mainfrom
tamirms/rocksdb-bulk-load

Conversation

@tamirms

@tamirms tamirms commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

The problem

The hot events index was written one RocksDB key per posting into every ledger's commit batch, about 6,000 keys per ledger on pubnet, and each key passed through the write-ahead log, the memtable and later compaction. Those keys were most of each commit. The next PR in this stack stops doing that and instead writes a whole slab of the index at once, hundreds of thousands of entries, which needs a way to add them to a column family without paying those three costs.

The idea

RocksDB can adopt a file you wrote yourself. Write the entries in sorted order as one SST file, RocksDB's own file format, and hand it over with IngestExternalFile: RocksDB links the file into the column family without passing a byte through the write-ahead log or the memtable. If the file's keys do not overlap any key the column family already holds, it goes straight to the bottom level and is never compacted. Readers see all of its entries at once, and they are durable when the call returns.

What the store adds

Store.LoadSorted(cf, entries) writes the file under the store's own loading/ directory with the column family's own block size, filter and compression, ingests it, and fails if the keys overlap rather than creating compaction work. It holds the store's lifecycle lock so Close waits for it, and a read-write open removes files a stopped process left behind. The per-column-family options are now kept by name so the file can be given them.

CFOptions.CacheIndexAndFilterBlocks moves each file's index and filter blocks from the heap into the block cache. Without it a column family made of many loaded files grows the heap by about 100 KB per file; with it the memory bound the next PR relies on holds.

Measured

On the stress stream of #1097, 109 loaded files all sat at the bottom level with level 0 empty. With the index and filter blocks in the cache, the table readers' heap went from 10.5 MB to 0.23 MB with lookups unchanged. The ingest call took 15 to 17 ms at p50 on macOS, which is the fsync.

🤖 Generated with Claude Code

@tamirms
tamirms added this pull request to stack #1089 October 6, 2026 09:22
@tamirms
tamirms removed this pull request from stack #1089 October 8, 2026 08:18
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch from c4d415b to 07d6a19 Compare October 8, 2026 08:18
@tamirms
tamirms changed the base branch from tamirms/events-cold-read-concurrency to tamirms/events-split-terms October 8, 2026 08:18
@tamirms
tamirms added this pull request to stack #1102 October 8, 2026 08:19
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch 3 times, most recently from b2642ed to 2bc993a Compare October 9, 2026 06:02
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch from 2bc993a to 1364aef Compare October 15, 2026 19:15
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch 3 times, most recently from e0521d0 to e8e8eba Compare October 21, 2026 09:12
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch 3 times, most recently from 4ba543f to e1db3b3 Compare October 21, 2026 11:16
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch 2 times, most recently from 14838ec to 02c898c Compare October 21, 2026 14:07
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch from 02c898c to cb03d81 Compare October 21, 2026 14:43
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch from cb03d81 to 6dee158 Compare October 21, 2026 17:05
@tamirms
tamirms added this pull request to stack #1157 October 27, 2026 10:21
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch from e38972b to e6f68c1 Compare October 28, 2026 08:05
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch from e6f68c1 to 376e227 Compare November 4, 2026 10:07
@tamirms
tamirms removed this pull request from stack #1157 November 4, 2026 10:20
@tamirms
tamirms added this pull request to stack #1184 November 4, 2026 10:21
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch 2 times, most recently from f23c104 to 2c011f5 Compare November 4, 2026 19:12
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch 2 times, most recently from 5243771 to eacb407 Compare November 12, 2026 09:17
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch 2 times, most recently from 245e535 to a6dd989 Compare November 12, 2026 13:32
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch from a6dd989 to 922187d Compare November 12, 2026 14:38
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch 2 times, most recently from 4e9d842 to ad0b768 Compare November 13, 2026 07:50
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch 2 times, most recently from 5961131 to c58f92e Compare November 13, 2026 11:38
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch from c58f92e to 71d0c3f Compare November 13, 2026 12:34
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch from 71d0c3f to 7170335 Compare November 13, 2026 12:54
Base automatically changed from tamirms/events-split-terms to main November 13, 2026 13:15
Store.LoadSorted writes an ascending sequence of entries as one SST file
under the store's own loading directory and moves it into the column
family with IngestExternalFile. Readers see all of the entries or none,
the entries survive a crash once the call returns, and they bypass the
WAL and the memtable. The keys must not overlap keys the column family
already holds: the file then lands below every other file and never
needs compacting, and an overlap fails the load. The store's lifecycle
lock is held for the whole load, so Close waits for it, and a read-write
open removes the files a stopped process left behind.

The per-column-family options are now kept by name, which the file
writer needs to give a file the column family's own block size, filter
and compression. CFOptions gains CacheIndexAndFilterBlocks, which keeps
an SST's index and filter blocks in the block cache instead of on the
heap, so a column family made of many loaded files cannot grow the heap
with them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BiCL19kUrno3Z4SAYgpAfh
@tamirms
tamirms force-pushed the tamirms/rocksdb-bulk-load branch from 7170335 to b8fce0e Compare November 13, 2026 13:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The sorted SST-loading change has no active review issues requiring resolution.

No open issues 🎉

What changed in this PR

Adds sorted SST-file loading to the RocksDB store, preparing for slab-based event indexing in #1097 without routing index entries through the write-ahead log or memtable.

Changes:

  • Introduces Store.LoadSorted with column-family-specific options, bottom-level placement checks, and lifecycle locking.
  • Cleans up unfinished load files on read-write open and adds configurable caching of index and filter blocks.
  • Adds tests for persistence, failure cleanup, shutdown coordination, column-family options, and cache memory use.
File Description
cmd/​stellar-rpc/​internal/​rpcv2/​rocksdb/​tuning.go Adds the index and filter block caching option.
cmd/​stellar-rpc/​internal/​rpcv2/​rocksdb/​rocksdb.go Retains options by column-family name, applies caching, and invokes startup cleanup.
cmd/​stellar-rpc/​internal/​rpcv2/​rocksdb/​load.go Implements sorted SST creation, ingestion, and unfinished-file cleanup.
cmd/​stellar-rpc/​internal/​rpcv2/​rocksdb/​load_test.go Tests loading behavior, lifecycle safety, recovery, and option handling.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants