Repository navigation
Conversation
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
from
October 8, 2026 08:18
c4d415b to
07d6a19
Compare
tamirms
changed the base branch from
tamirms/events-cold-read-concurrency
to
tamirms/events-split-terms
October 8, 2026 08:18
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
3 times, most recently
from
October 9, 2026 06:02
b2642ed to
2bc993a
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
from
October 15, 2026 19:15
2bc993a to
1364aef
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
3 times, most recently
from
October 21, 2026 09:12
e0521d0 to
e8e8eba
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
3 times, most recently
from
October 21, 2026 11:16
4ba543f to
e1db3b3
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
2 times, most recently
from
October 21, 2026 14:07
14838ec to
02c898c
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
from
October 21, 2026 14:43
02c898c to
cb03d81
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
from
October 21, 2026 17:05
cb03d81 to
6dee158
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
from
October 28, 2026 08:05
e38972b to
e6f68c1
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
from
November 4, 2026 10:07
e6f68c1 to
376e227
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
2 times, most recently
from
November 4, 2026 19:12
f23c104 to
2c011f5
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
2 times, most recently
from
November 12, 2026 09:17
5243771 to
eacb407
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
2 times, most recently
from
November 12, 2026 13:32
245e535 to
a6dd989
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
from
November 12, 2026 14:38
a6dd989 to
922187d
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
2 times, most recently
from
November 13, 2026 07:50
4e9d842 to
ad0b768
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
2 times, most recently
from
November 13, 2026 11:38
5961131 to
c58f92e
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
from
November 13, 2026 12:34
c58f92e to
71d0c3f
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
from
November 13, 2026 12:54
71d0c3f to
7170335
Compare
Store.LoadSorted writes an ascending sequence of entries as one SST file under the store's own loading directory and moves it into the column family with IngestExternalFile. Readers see all of the entries or none, the entries survive a crash once the call returns, and they bypass the WAL and the memtable. The keys must not overlap keys the column family already holds: the file then lands below every other file and never needs compacting, and an overlap fails the load. The store's lifecycle lock is held for the whole load, so Close waits for it, and a read-write open removes the files a stopped process left behind. The per-column-family options are now kept by name, which the file writer needs to give a file the column family's own block size, filter and compression. CFOptions gains CacheIndexAndFilterBlocks, which keeps an SST's index and filter blocks in the block cache instead of on the heap, so a column family made of many loaded files cannot grow the heap with them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BiCL19kUrno3Z4SAYgpAfh
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
from
November 13, 2026 13:16
7170335 to
b8fce0e
Compare
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The sorted SST-loading change has no active review issues requiring resolution.
No open issues 🎉
What changed in this PR
Adds sorted SST-file loading to the RocksDB store, preparing for slab-based event indexing in #1097 without routing index entries through the write-ahead log or memtable.
Changes:
- Introduces
Store.LoadSortedwith column-family-specific options, bottom-level placement checks, and lifecycle locking. - Cleans up unfinished load files on read-write open and adds configurable caching of index and filter blocks.
- Adds tests for persistence, failure cleanup, shutdown coordination, column-family options, and cache memory use.
| File | Description |
|---|---|
cmd/stellar-rpc/internal/rpcv2/rocksdb/tuning.go |
Adds the index and filter block caching option. |
cmd/stellar-rpc/internal/rpcv2/rocksdb/rocksdb.go |
Retains options by column-family name, applies caching, and invokes startup cleanup. |
cmd/stellar-rpc/internal/rpcv2/rocksdb/load.go |
Implements sorted SST creation, ingestion, and unfinished-file cleanup. |
cmd/stellar-rpc/internal/rpcv2/rocksdb/load_test.go |
Tests loading behavior, lifecycle safety, recovery, and option handling. |
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The hot events index was written one RocksDB key per posting into every ledger's commit batch, about 6,000 keys per ledger on pubnet, and each key passed through the write-ahead log, the memtable and later compaction. Those keys were most of each commit. The next PR in this stack stops doing that and instead writes a whole slab of the index at once, hundreds of thousands of entries, which needs a way to add them to a column family without paying those three costs.
The idea
RocksDB can adopt a file you wrote yourself. Write the entries in sorted order as one SST file, RocksDB's own file format, and hand it over with
IngestExternalFile: RocksDB links the file into the column family without passing a byte through the write-ahead log or the memtable. If the file's keys do not overlap any key the column family already holds, it goes straight to the bottom level and is never compacted. Readers see all of its entries at once, and they are durable when the call returns.What the store adds
Store.LoadSorted(cf, entries)writes the file under the store's ownloading/directory with the column family's own block size, filter and compression, ingests it, and fails if the keys overlap rather than creating compaction work. It holds the store's lifecycle lock soClosewaits for it, and a read-write open removes files a stopped process left behind. The per-column-family options are now kept by name so the file can be given them.CFOptions.CacheIndexAndFilterBlocksmoves each file's index and filter blocks from the heap into the block cache. Without it a column family made of many loaded files grows the heap by about 100 KB per file; with it the memory bound the next PR relies on holds.Measured
On the stress stream of #1097, 109 loaded files all sat at the bottom level with level 0 empty. With the index and filter blocks in the cache, the table readers' heap went from 10.5 MB to 0.23 MB with lookups unchanged. The ingest call took 15 to 17 ms at p50 on macOS, which is the fsync.
🤖 Generated with Claude Code