Repository navigation
Conversation
0718fc5 to
c2bdf2c
Compare
c2bdf2c to
c4b50ca
Compare
3b1e4a5 to
0e15b2e
Compare
0e15b2e to
d6c50bf
Compare
d6c50bf to
96ad1f7
Compare
96ad1f7 to
b5e421f
Compare
05254eb to
7a0e4bf
Compare
7a0e4bf to
177b87c
Compare
1224a78 to
a021c35
Compare
a021c35 to
5d47c3b
Compare
b1ee01a to
98dd32e
Compare
98dd32e to
1f3c9de
Compare
5f6f2fe to
b994cae
Compare
b994cae to
73c7511
Compare
73c7511 to
163f4ee
Compare
163f4ee to
6f995f0
Compare
The hot events index held every term of a chunk as an in-memory bitmap and wrote one events_index key per posting into each ledger's batch. Its memory grew with the number of distinct terms in the chunk, and the per-posting keys were most of each commit. Event ids are now cut into slabs of 65,536, the matcher's unit. The slab being filled is a fixed array of postings with per-bucket chains, which one writer appends to and readers walk without a lock. When the next slab starts, the full one is sealed in the background: its terms are written in key order as one file and loaded into events_index under slab || term. events_index has auto compaction off, and its files never overlap so compaction would have nothing to do; their index blocks live in the block cache. The writer waits for the previous seal before it starts the next one, so the index holds at most two slabs in memory whatever the chunk holds, and the ledger commit carries no index bytes any more. A failed seal keeps its slab readable from memory and fails the next ledger's apply, through the apply hook, which can now fail; the ledger report then names PhaseApply, and the write phases carry their item counts only once the commit landed. The next open indexes the unsealed events again from events_data, through the same path ingestion uses, and seals a full slab among them as soon as the next one starts. LookupKeys reads the slabs the window touches, from memory or from events_index, and returns a bitmap per key, empty rather than nil, since it has not seen the rest of the chunk. The bench sink counts an ingest total only for a ledger whose apply succeeded. The design documents describe the index as it is now. Measured with bench-ingest hot on the same synthetic ledgers (6,000 events of 7 terms each per ledger) at a 600 ms cadence, 600 ledgers, two runs each: per-ledger ingest p50 139 and 144 ms before, 70 and 71 ms after; p99 288 and 319 ms before, 82 and 88 ms after; the events, commit and apply phases went from 12, 72 and 11 ms to 4, 21 and 1 ms at p50; peak RSS 1,373 and 1,379 MB before, 549 and 545 MB after. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BiCL19kUrno3Z4SAYgpAfh
The live slab buckets postings by the leading 16 bits of the term key, which anyone can compute, so a caller could grind topic values into the bucket of a term every event carries and make each lookup walk that whole chain. The hot index now files every term under routedKey(ColdIndexSecret(catalogSecret, chunkID), term), the key the cold build gives it, so a term's bucket cannot be predicted and hot keys equal cold keys. A lookup blinds each term once for every slab. Blinded keys still ascend by bucket, so the seal needs no sort, and IndexCF keys keep their slab prefix. event.NewWithStore and the hotchunk openers take the catalog secret. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wwsd8qTEGeSb7poFEYcgg9
A hot lookup probes every sealed slab its window touches for every term, and a slab that lacks the term cost a block read to say so. IndexCF now has a 10-bit-per-key bloom filter. The seal's file writer copies the column family's options, so every sealed file carries it, and its filter blocks live in the block cache beside the index blocks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wwsd8qTEGeSb7poFEYcgg9
The open-serving test came in with the rebase onto feature/full-history and still called hotchunk.OpenReadyWrite without the secret this PR adds, so the package's tests did not build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The hot write path derived each event's terms into a slice of its own and, after the ledger committed, routed every posting with SipHash on the ingest goroutine before adding it to the index. It now derives a ledger's terms into one reused buffer and routes them on another goroutine while the batch is written and committed, so the post-commit apply only appends postings. Each event's data key reuses one buffer too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjQed1VX9uGU9PWpAjBiz7
6f995f0 to
fbe03f6
Compare
|
I think the live slab would be simpler as a map of term to bitmap instead of the posting list. The seal would stay as it is: a full slab is still written as one sorted file of (slab, term) -> bitmap and loaded with IngestExternalFile, so the index still skips the WAL, the memtable and compaction. The only difference is that the seal sorts the map's terms first. The posting list needs hash buckets, and the buckets are the reason for the blinded keys, the catalog secret in every hotchunk opener and the routing goroutine. A Go map needs none of these. Live lookups also get cheaper: they copy a ready bitmap instead of walking a chain per query. Memory can be larger than the posting list, but it stays bounded, because a slab holds at most 458,752 terms. The cost is the update. The description says one in-memory bitmap per slab was about five times slower to update. That is about 6,000 in-memory adds per ledger, so I expect the difference to be small against the 10 ms p50. Was that a bitmap per term, and how many ms per ledger is it? |
|
Thanks. Taking simplicity first, since that's the main point. The map would remove about 80 lines here: the 110-line posting list and the 25 lines that route terms on a second goroutine would become about 50 lines of map code. It wouldn't remove the blinding or the secret, though. The freeze hands each sealed slab to the cold build as a run as it is, which works because the hot index files terms under the cold build's blinded keys, in that order. So with a map the seal would still blind and sort, and the openers keep the secret. The posting list is also the cold build's slab buffer in #1098, so the map would replace it there too, and backfill would take on the costs below. The design ranked p99 ingestion time first, memory second and query latency last, so here is the map against each. This is the alternative the description lists as "one in-memory bitmap per slab". I had measured it only as the index update on its own, so I've rerun it end to end: a prototype of your exact design (the live slab as a map from term to roaring bitmap, no routing goroutine, blinding at the seal) against this PR with Ingestion. Per ledger, p50 / p99, averaged over the runs:
At pubnet density the difference is within the noise between runs. At sac-9000, phase 3's stress profile, it isn't. The map's index update takes about 8 ms more on the median ledger and 13 ms more at p99. The whole ledger takes about 8 ms more at p50 and 7 ms more at p99, which is 7% of the 105 ms that phase 3 allows a ledger at p99. Both designs are over that budget here because this PR comes before the latency work tracked in #1148. Memory. One live slab:
At peak, for the slab buffers alone:
At typical density the two are about the same size. The gap is the worst case, which live traffic can force with unique topic values, and which backfill would multiply by its workers. The collector's cost is small at these densities. The posting list is two arrays without pointers, so the collector never scans it, while a map's entries are objects it marks on every cycle. Over 1,000 sac-9000 ledgers the collector used 4.8 s of CPU with the posting list and 5.5 s with the map, and over 3,000 pubnet ledgers 2.2 s and 4.2 s. Query latency. Here the map is much better. A live lookup of a term in every event of the slab copies one bitmap in 0.15 µs, where the posting list walks its chain in about 380 µs. But query latency came last in this design. So I'd keep the posting list. Stress density is where phase 3's budget bites. There the posting list keeps about 8 ms off the median ledger and 7 ms off the p99, and its memory peak doesn't depend on traffic. That's worth more than the 80 lines the map would save. Edit: I first ran this at 6,000 transactions per ledger, but phase 3's stress profile is 9,000, so all the numbers above are now from sac-9000 on the phase 3 machine. |
Several parts of the hot index were written out where they are used, so a reader had to work out what each one does at every place it appears: - The walk of a hash bucket's chain, with its 1-based links, appeared in both bitmap and terms. It is now one iterator, chain. - The seal's outcome lived in a one-slot channel that add, rotate and settle each took from and put back. A sealer now does that behind failed, wait and start. - HotStore routed a ledger's terms through a channel field, a goroutine and two waits. That is now ledgerTerms.routeAsync and waitRouted, and reset waits for the routing itself, so no caller has to remember to. - lookup chose between memory and IndexCF inside its loop; slabBitmap names that choice. Dense comments and design-doc sections (8.2, 12 and 15) are rewritten as short sentences, lists and a table. Behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seal built a new IndexCF key for every term of the slab. LoadSorted writes each entry before it pulls the next, so one key buffer serves the whole slab, and a seal no longer allocates a key per term. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The new index-key check prevents existing hot databases from reopening after upgrade without format versioning and migration safeguards, while recovery and retained buffers pose excessive memory risks.
3 open issues
What changed in this PR
This PR replaces rpcv2’s chunk-wide in-memory hot events index with slab-based indexing, aiming to reduce memory use and remove index writes from each ledger commit.
Changes:
- Seals 65,536-event slabs into RocksDB sorted files and serves lookups across sealed and live slabs.
- Passes the catalog secret through hot-store openers to blind index keys consistently with the cold index.
- Updates recovery, failure reporting, tests, benchmarks, and design documentation for slab indexing.
| File | Description |
|---|---|
design-docs/getevents-full-history-design.md |
Documents slab storage, queries, recovery, and memory estimates. |
design-docs/full-history-streaming-workflow.md |
Updates ingestion ownership and durability documentation. |
cmd/stellar-rpc/internal/rpcv2/stores/hotchunk/hotchunk.go |
Passes catalog secrets and propagates post-commit apply errors. |
cmd/stellar-rpc/internal/rpcv2/stores/hotchunk/hotchunk_test.go |
Tests index options, recovery, and failed-apply reporting. |
cmd/stellar-rpc/internal/rpcv2/stores/event/slab_match.go |
Updates comments about bitmap ownership. |
cmd/stellar-rpc/internal/rpcv2/stores/event/slab_match_test.go |
Adapts fixtures to the replacement index. |
cmd/stellar-rpc/internal/rpcv2/stores/event/roaring_contract_test.go |
Refocuses bitmap tests on read-only inputs. |
cmd/stellar-rpc/internal/rpcv2/stores/event/reader.go |
Documents window coverage and empty-result semantics. |
cmd/stellar-rpc/internal/rpcv2/stores/event/matches_differential_test.go |
Checks query results during concurrent slab ingestion. |
cmd/stellar-rpc/internal/rpcv2/stores/event/match_test.go |
Updates miss and collision tests; removes shared-mirror tests. |
cmd/stellar-rpc/internal/rpcv2/stores/event/index.go |
Adds append-based term extraction for buffer reuse. |
cmd/stellar-rpc/internal/rpcv2/stores/event/index_test.go |
Uses the shared maximum-term constant. |
cmd/stellar-rpc/internal/rpcv2/stores/event/hot_warmup_test.go |
Updates recovery and corruption checks for slabs. |
cmd/stellar-rpc/internal/rpcv2/stores/event/hot_store.go |
Integrates slab indexing, term routing, and recovery. |
cmd/stellar-rpc/internal/rpcv2/stores/event/hot_store_test.go |
Adapts ingestion and lookup tests to slab storage. |
cmd/stellar-rpc/internal/rpcv2/stores/event/hot_index.go |
Implements live postings, background sealing, and slab lookups. |
cmd/stellar-rpc/internal/rpcv2/stores/event/hot_index_test.go |
Tests sealing, concurrent reads, key blinding, and recovery. |
cmd/stellar-rpc/internal/rpcv2/stores/event/concurrent_bitmaps.go |
Removes the previous in-memory index implementation. |
cmd/stellar-rpc/internal/rpcv2/stores/event/concurrent_bitmaps_writer_test.go |
Removes tests specific to the retired writer. |
cmd/stellar-rpc/internal/rpcv2/stores/event/concurrent_bitmaps_test.go |
Removes tests specific to the retired bitmap map. |
cmd/stellar-rpc/internal/rpcv2/stores/event/bitmaps.go |
Removes obsolete concurrent-index guidance from comments. |
cmd/stellar-rpc/internal/rpcv2/stores/event/benchmark_test.go |
Benchmarks the slab index using RocksDB storage. |
cmd/stellar-rpc/internal/rpcv2/rpcv2test/rpcv2test.go |
Supplies catalog secrets when seeding hot chunks. |
cmd/stellar-rpc/internal/rpcv2/query/tx_lookup_test.go |
Updates hot-store test openers. |
cmd/stellar-rpc/internal/rpcv2/query/resolve_test.go |
Updates query-resolution test openers. |
cmd/stellar-rpc/internal/rpcv2/query/registry.go |
Supplies the secret when reopening serving handles. |
cmd/stellar-rpc/internal/rpcv2/query/registry_test.go |
Updates registry test openers. |
cmd/stellar-rpc/internal/rpcv2/query/ledger_scan_test.go |
Updates the hot/cold boundary fixture opener. |
cmd/stellar-rpc/internal/rpcv2/query/event_page_test.go |
Supplies secrets for event-pagination fixtures. |
cmd/stellar-rpc/internal/rpcv2/progress.go |
Passes the secret to the progress-probe opener. |
cmd/stellar-rpc/internal/rpcv2/open_serving_test.go |
Updates the handle-closure verification opener. |
cmd/stellar-rpc/internal/rpcv2/lifecycle/lifecycle_test.go |
Updates lifecycle test openers. |
cmd/stellar-rpc/internal/rpcv2/lifecycle/helpers_test.go |
Supplies the secret in the hot-chunk helper. |
cmd/stellar-rpc/internal/rpcv2/ingest/service.go |
Preserves committed item counts when apply fails. |
cmd/stellar-rpc/internal/rpcv2/ingest/metrics.go |
Documents post-commit failure metric semantics. |
cmd/stellar-rpc/internal/rpcv2/ingest/ingest_test.go |
Adds failed-apply metric coverage and updates openers. |
cmd/stellar-rpc/internal/rpcv2/hotloop.go |
Passes secrets when creating or resuming hot chunks. |
cmd/stellar-rpc/internal/rpcv2/hotloop_test.go |
Updates ingestion-loop test openers. |
cmd/stellar-rpc/internal/rpcv2/e2e_test.go |
Updates the end-to-end reopen call. |
cmd/stellar-rpc/internal/rpcv2/bench/csvsink.go |
Excludes failed applies from successful ingest totals. |
cmd/stellar-rpc/internal/rpcv2/bench/csvsink_test.go |
Tests exclusion of failed-apply totals. |
cmd/stellar-rpc/internal/rpcv2/backfill/process.go |
Supplies the secret when opening hot freeze sources. |
Review effort: Balanced
| if len(last) != hotIndexKeyLen { | ||
| return 0, fmt.Errorf("events: unexpected %s key length %d (want %d)", IndexCF, len(last), hotIndexKeyLen) | ||
| } | ||
| return uint32(binary.BigEndian.Uint16(last)) + 1, nil |
There was a problem hiding this comment.
rpcv2 hasn't been released yet, and until GA operators re-ingest instead of upgrading hot DBs. The demotion path in the workflow doc covers hot formats that a release has shipped: release 1 writes bare state values, and the format grammar ships with the first bump after it. A hot DB from before this PR fails to open with the key length error, which is the outcome we want before GA.
| return func() error { | ||
| h.terms.waitRouted() | ||
| return h.applyLedger(startID, &h.terms) | ||
| }, nil |
There was a problem hiding this comment.
This is bounded by one ledger, not by the chunk or the number of terms: at most 116 bytes per event of the largest ledger the handle ingested (7 terms of 16 bytes, plus a 4-byte end offset). The handle needed that much to ingest that ledger, so keeping it doesn't raise its peak. After the handoff, the old handle holds it until it closes, so the total is at most two ledgers' worth. Releasing it would mean allocating it again for every large ledger, on the ingestion path, so I'm keeping the reuse.
| // Collect the terms first and index them after the scan: indexing can | ||
| // seal, and a seal takes the lifecycle lock the scan holds. | ||
| var terms ledgerTerms | ||
| for entry, err := range chunkStore.IterateRange(DataCF, encodeDataKey(from), encodeDataKey(total-1)) { |
There was a problem hiding this comment.
The unsealed tail is bounded: the slab being sealed, the live slab, and the one ledger whose apply was interrupted. An apply error stops the hot loop, and an open whose replay fails commits nothing, so the tail can't grow across restarts. Replay holds at most 116 bytes per event of that tail, about 15 MB for the two slabs plus the interrupted ledger's terms, so I'm keeping the single scan.


The problem
Indexing events for getEvents was both the biggest memory risk in ingestion and most of the cost of every ledger commit.
The hot events index kept one bitmap per term (every contract id, topic and event type seen in the chunk) in memory, listing the events that carry it, for the whole chunk, in every open hot database. That grows with the number of distinct terms, and a chunk can hold tens of millions of them under adversarial traffic, so there was no bound. And every ledger's commit wrote one RocksDB key per posting (one term on one event), about 6,000 keys per ledger on pubnet, through the write-ahead log, the memtable and later compaction. Those keys were most of each commit.
The idea
Keep the index out of RocksDB's write path entirely, and keep only a fixed-size slice of it in memory.
Event ids are cut into slabs of 65,536. Postings for the slab being filled go into a fixed in-memory buffer. When the slab is full, it is sealed: its terms are written in key order as one SST file, RocksDB's own sorted file format, and handed to RocksDB with
IngestExternalFile, which adds the file to the index column family as it is. Every key in the file starts with the slab number, so no two slabs overlap, and RocksDB places each file straight at the bottom level. Nothing goes through the write-ahead log or the memtable, and nothing is ever flushed or compacted. Readers see the whole slab at once, and it is durable the moment the call returns.Why ingestion gets faster
The ledger commit carries no index bytes at all. A ledger's terms are derived into one reused buffer, and another goroutine routes them to their keys while the ledger's batch is written and committed, so indexing the ledger after the commit is appending its postings to the in-memory buffer. The only disk work for the index is one sequential file write per 65,536 events, done in the background.
Why memory is bounded
A slab is 65,536 events and an event carries at most 7 terms, so the live buffer holds at most 458,752 postings, whatever the number of distinct terms in the chunk or events in a ledger. At most two slabs are in memory, the one being filled and the one being sealed: the writer waits for the previous seal before starting the next, so a slow disk stalls ingestion at a slab boundary instead of letting slabs pile up. The sealed files' index and filter blocks live in RocksDB's block cache, which has a fixed size, not on the heap.
Blinded keys
The slab being filled groups its postings by the leading 16 bits of each term's key. Anyone can compute an unblinded key, so a caller could pick topic values whose keys land in the group of a term every event carries, and every lookup of those topics would walk that whole group. The hot index files each term under the key the cold build gives it: the term's key blinded with the chunk's cold index secret, which derives from the node's random catalog secret. A term's group can't be predicted, and a term has the same key in the hot and the cold index. Blinding costs one SipHash per posting at ingest, about 44 ns, and it runs on the routing goroutine while the ledger commits.
event.NewWithStoreand the hotchunk openers take the catalog secret.What it costs
Queries read sealed slabs from RocksDB instead of from memory, which is slower than the in-memory index. Each sealed file carries a bloom filter of 10 bits per key, so a slab that lacks a term answers without a block read. Query latency was the lowest priority here, after memory and ingestion latency.
Failure and recovery
A seal runs in the background, so its failure is reported through ingestion. If a seal fails, the slab's postings are still in memory, so queries keep working, and the next ledger's apply returns the error, so ingestion stops rather than continuing with a slab missing from the index. On restart, the slabs that were not sealed are indexed again from the stored event data and sealed again.
Alternatives measured
The engine in #902 bounds memory only as a function of ledger density and term count, and spends memory on query speed. One in-memory bitmap per slab is bounded but about five times slower to update. Writing sealed slabs through RocksDB's normal write path stalls commits on flushes, and term-first keys need compaction. The sealed-slab file was the only option that is both bounded by construction and faster to ingest.
Results
bench ingest hoton a real pubnet chunk of 3,000 ledgers, with its diagnostic events: ingest per ledger went from 19 to 10 ms at p50 and from 39 to 16 ms at p99. A kill -9 harness that ingests in a child process, kills it at a random moment, reopens and verifies every term passed 100 of 100 rounds, a third of them with the kill landing during a seal.🤖 Generated with Claude Code