Repository navigation
events, deps: memory-map the cold MPHF and overlap cold reads, roaring v2.26.0 - #1086
Merged
Merged
Conversation
tamirms
force-pushed
the
tamirms/roaring-2.26-mphf-mmap
branch
from
October 8, 2026 08:18
1ac8a61 to
4f31522
Compare
tamirms
force-pushed
the
tamirms/roaring-2.26-mphf-mmap
branch
from
October 8, 2026 08:34
4f31522 to
c8aac0c
Compare
tamirms
force-pushed
the
tamirms/roaring-2.26-mphf-mmap
branch
2 times, most recently
from
October 9, 2026 06:02
44efd6d to
193bda0
Compare
tamirms
force-pushed
the
tamirms/roaring-2.26-mphf-mmap
branch
2 times, most recently
from
October 15, 2026 19:15
1d0ed30 to
3bdacff
Compare
This was referenced Oct 19, 2026
tamirms
force-pushed
the
tamirms/roaring-2.26-mphf-mmap
branch
2 times, most recently
from
October 21, 2026 09:12
0f5c469 to
e66bb2c
Compare
tamirms
force-pushed
the
tamirms/roaring-2.26-mphf-mmap
branch
2 times, most recently
from
October 21, 2026 09:52
22d71a7 to
a434820
Compare
tamirms
force-pushed
the
tamirms/roaring-2.26-mphf-mmap
branch
from
October 21, 2026 11:16
a434820 to
a890554
Compare
tamirms
force-pushed
the
tamirms/roaring-2.26-mphf-mmap
branch
4 times, most recently
from
October 21, 2026 17:05
e194ca3 to
1341976
Compare
tamirms
marked this pull request as ready for review
October 21, 2026 17:12
tamirms
force-pushed
the
tamirms/roaring-2.26-mphf-mmap
branch
from
October 21, 2026 17:15
1341976 to
fb3ff5f
Compare
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The unreleased Roaring fork and mapped concurrent-read changes warrant human validation of compatibility and lifecycle safety.
No open issues 🎉
What changed in this PR
Improves cold-event query performance by memory-mapping indexes, batching term lookups, and overlapping packfile reads.
Changes:
- Replaces whole-index reads with memory mapping and batched lookups.
- Enables up to eight concurrent cold-event reads.
- Updates Roaring dependencies with a temporary fork pin and adds bitmap contract tests.
| File | Description |
|---|---|
| go.sum | Updates dependency checksums. |
| go.mod | Updates Roaring and bitset; pins the temporary Roaring fork. |
| cmd/stellar-rpc/internal/rpcv2/stores/event/roaring_contract_test.go | Tests bitmap ownership, searches, concurrency, and allocations. |
| cmd/stellar-rpc/internal/rpcv2/stores/event/cold_reader.go | Uses batched term lookups. |
| cmd/stellar-rpc/internal/rpcv2/stores/event/cold_reader_test.go | Updates comments for mapped-index deletion behavior. |
| cmd/stellar-rpc/internal/rpcv2/stores/event/cold_index.go | Documents batched lookup behavior. |
| cmd/stellar-rpc/internal/rpcv2/stores/event/cold_format.go | Adds mapped index opening and batch lookup support. |
| cmd/stellar-rpc/internal/rpcv2/stores/event/cold_format_test.go | Checks batch and individual lookup equivalence. |
| cmd/stellar-rpc/internal/rpcv2/query/resolve.go | Enables eight concurrent cold-event reads. |
Review effort: Balanced
tamirms
force-pushed
the
tamirms/roaring-2.26-mphf-mmap
branch
from
October 22, 2026 18:41
fb3ff5f to
879a0f4
Compare
tamirms
force-pushed
the
tamirms/roaring-2.26-mphf-mmap
branch
from
October 22, 2026 19:46
879a0f4 to
3e305ba
Compare
tamirms
force-pushed
the
tamirms/roaring-2.26-mphf-mmap
branch
3 times, most recently
from
October 23, 2026 06:14
b35752b to
a42a881
Compare
…g v2.26.0 Each cold events read opened its chunk's index.hash by reading the whole file into memory. It now memory-maps the file instead. The pages come from the kernel's page cache, which every reader of the chunk shares, so an open costs a map plus the pages its lookups touch, rather than a copy of a file that grows with the chunk's term count. LookupKeys now resolves all of a query's terms with one streamhash QueryBatch call (stellar/streamhash#24) instead of one lookup per term. On a cold page cache a lookup reads a few index.hash pages one after another; QueryBatch requests every term's pages together, so a query's lookups cost about one round of reads instead of one per term. On gp3 with the file evicted from the page cache, opening a 60M-term index and looking up 15 terms took 2.5 ms instead of 17.3 ms at p50, and 5 terms 2.3 ms instead of 6.8 ms. Cold events readers were opened with a concurrency of 0, so each lookup and fetch issued its index.pack and events.pack reads one after another: one per term for a lookup, and one per batch of a page's scattered events for a fetch. The resolver now opens them with a concurrency of 8, which packfile's ReadItems uses for both packs. Measured on gp3 with a cold page cache, with index.pack's big terms split by slab (a later change in this stack), lookups of several terms got 40 to 62% faster at p50: five popular terms took 1.7 ms instead of 4.4 ms on pubnet chunk 6465, and 1.9 ms instead of 4.6 ms on a chunk folded to seven times its density. Single-term lookups, and lookups on a volume already at its throughput limit, are unchanged. roaring moves from v2.18.2 to v2.26.0, which adds vectorized container kernels. Its serialized form is roaring's portable format, so existing artifacts read the same. go.mod replaces it with the head of RoaringBitmap/roaring#572, which sits on RoaringBitmap/roaring#566, until a release includes both: they count or test before allocating in FastAnd and in run-container intersections, so an intersection that comes out empty allocates nothing. Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt Claude-Session: https://claude.ai/code/session_01JCzMKKrK3NstGKvaxsHhL2 Claude-Session: https://claude.ai/code/session_01Kkry5utNweWZqUKL88vJ2X Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new head of RoaringBitmap/roaring#572 looks up a one-key call's key in every input, up to eight, before it allocates anything. So an empty per-slab intersection allocates only its result bitmap, whatever mix of array, bitmap and run containers its terms hold. roaring_contract_test.go pins the roaring properties the event index relies on, written against roaring alone so it runs first on any version bump: FastAnd and the aggregations read their arguments and return containers that share no storage with them, the value searches are read-only and inclusive, and an empty FastAnd allocates only its result. That last test makes the one-key call with every mix of array, bitmap and run terms up to four terms and a sample of mixes up to seven, on a whole and a partial slab, with the result emptied at each term in turn. It fails on the previous pin, 64285f3, in 3,263 of its 3,727 cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wwsd8qTEGeSb7poFEYcgg9
tamirms
force-pushed
the
tamirms/roaring-2.26-mphf-mmap
branch
from
October 23, 2026 07:09
a42a881 to
dd4d5dc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Cold events readers are opened per request, and each open read its chunk's whole
index.hashinto memory, however few terms the request looked up. The file grows with the chunk's term count: 0.34 MB on pubnet chunk 6504, 0.94 MB on K7 and 18.5 MB on sac-6000. Each open copied the whole file into the heap, and on a cold page cache read all of it from the volume, whose 125 MiB/s baseline the whole node shares. On sac-6000, that caps the whole node at about seven cold opens a second.The readers were also opened with a concurrency of 0, so each lookup and fetch issued its index.pack and events.pack reads one after another: one per term for a lookup, and one per batch of a page's scattered events for a fetch.
What
index.hash(streamhash.Open) instead of reading it whole (os.ReadFile+streamhash.OpenBytes), so a request reads only the pages its lookups touch.LookupKeyslooks up all of a query's terms with oneQueryBatchcall (Add QueryBatch, to look up many keys at once streamhash#24), which requests every term's pages before it reads any. One lookup per term would fault each term's pages in series.QueryBatchneeds a mapped index, which is why it lands here.ReadItemsoverlaps a request's index.pack and events.pack reads.roaring_contract_test.gotests the roaring properties the event index relies on.Measurements
Hash index, on gp3. Each trial waits a second with the volume idle, evicts
index.hashfrom the page cache, then times opening it, looking up the terms and closing it. Before is oneLookupper term on the file read whole, and after is oneLookupBatchon the mapped file. p50 of 40 rounds, alternating before and after on the same real terms. Read is the trial'sread_bytesfrom/proc/self/io.K7 is 70,000 real pubnet ledgers folded into one chunk, seven per ledger. sac-6000 is synthetic, with 6,000 SAC transfers per ledger, each to a new account.
The mapped open reads 128 KiB, the kernel's read-around for the first page it touches (
read_ahead_kbis 128 on this volume), and each term adds a few KiB. So the gain is smallest on pubnet, whose whole file is 332 KiB. Batching is what keeps extra terms cheap: in an earlier run, the mapped file looked up one term at a time took 17.3 ms for 15 terms on sac-6000.Concurrency, on gp3 with a cold page cache and #1082 applied: a lookup of five popular terms took 1.7 ms instead of 4.4 ms on pubnet chunk 6465, and 1.9 ms instead of 4.6 ms on K7. Single-term lookups, and lookups on a volume already at its throughput limit, are unchanged.
roaring
go.modrequires v2.26.0 and replaces it with the head of RoaringBitmap/roaring#572, which sits on RoaringBitmap/roaring#566 and on roaring's master. The two makeFastAndand run-container intersections count or test before allocating, so a plan that matches nothing in a slab allocates only its empty result. #1087's slab walk runs that intersection on every slab..golangci.ymlallows this one replace.roaring_contract_test.gotests roaring alone, so a version bump that breaks a property the event index relies on fails there first:FastAndand the aggregations return containers that share no storage with their arguments, the value searches are read-only and inclusive, and an emptyFastAndallocates only its result. The last test covers every mix of array, bitmap and run terms up to four, and a sample up to seven. On v2.26.0 without the replace, it fails all 3,727 cases.Known limitations
ReadItemsruns its callback on worker goroutines, where a panic ends the process instead of failing the request. events: turn panics in cold-read goroutines into errors #1126 turns those panics into errors.🤖 Generated with Claude Code