Skip to content

events, deps: memory-map the cold MPHF and overlap cold reads, roaring v2.26.0 - #1086

Merged
tamirms merged 2 commits into
mainfrom
tamirms/roaring-2.26-mphf-mmap
Oct 24, 2026
Merged

tamirms merged 2 commits into
mainfrom
tamirms/roaring-2.26-mphf-mmap

Conversation

@tamirms

@tamirms tamirms commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Why

Cold events readers are opened per request, and each open read its chunk's whole index.hash into memory, however few terms the request looked up. The file grows with the chunk's term count: 0.34 MB on pubnet chunk 6504, 0.94 MB on K7 and 18.5 MB on sac-6000. Each open copied the whole file into the heap, and on a cold page cache read all of it from the volume, whose 125 MiB/s baseline the whole node shares. On sac-6000, that caps the whole node at about seven cold opens a second.

The readers were also opened with a concurrency of 0, so each lookup and fetch issued its index.pack and events.pack reads one after another: one per term for a lookup, and one per batch of a page's scattered events for a fetch.

What

  • The cold events reader memory-maps index.hash (streamhash.Open) instead of reading it whole (os.ReadFile + streamhash.OpenBytes), so a request reads only the pages its lookups touch.
  • LookupKeys looks up all of a query's terms with one QueryBatch call (Add QueryBatch, to look up many keys at once streamhash#24), which requests every term's pages before it reads any. One lookup per term would fault each term's pages in series. QueryBatch needs a mapped index, which is why it lands here.
  • The query resolver opens cold events readers with a concurrency of 8, so packfile's ReadItems overlaps a request's index.pack and events.pack reads.
  • roaring moves from v2.18.2 to the head of fastaggregation: count before allocating in FastAnd RoaringBitmap/roaring#572, and roaring_contract_test.go tests the roaring properties the event index relies on.

Measurements

Hash index, on gp3. Each trial waits a second with the volume idle, evicts index.hash from the page cache, then times opening it, looking up the terms and closing it. Before is one Lookup per term on the file read whole, and after is one LookupBatch on the mapped file. p50 of 40 rounds, alternating before and after on the same real terms. Read is the trial's read_bytes from /proc/self/io.

index.hash query terms before p50 after p50 before read after read
pubnet chunk 6504 (0.34 MB, 1.1M terms) 1 2.29 ms 1.54 ms 332 KiB 132 KiB
5 2.36 ms 1.77 ms 332 KiB 144 KiB
15 2.42 ms 1.91 ms 332 KiB 200 KiB
K7 (0.94 MB, 3.1M terms) 1 4.32 ms 1.70 ms 924 KiB 132 KiB
5 4.28 ms 1.91 ms 924 KiB 152 KiB
15 4.28 ms 2.00 ms 924 KiB 208 KiB
sac-6000 (18.5 MB, 60M terms) 1 58.00 ms 1.99 ms 18,032 KiB 132 KiB
5 57.99 ms 2.58 ms 18,032 KiB 160 KiB
15 55.66 ms 2.80 ms 18,032 KiB 232 KiB

K7 is 70,000 real pubnet ledgers folded into one chunk, seven per ledger. sac-6000 is synthetic, with 6,000 SAC transfers per ledger, each to a new account.

The mapped open reads 128 KiB, the kernel's read-around for the first page it touches (read_ahead_kb is 128 on this volume), and each term adds a few KiB. So the gain is smallest on pubnet, whose whole file is 332 KiB. Batching is what keeps extra terms cheap: in an earlier run, the mapped file looked up one term at a time took 17.3 ms for 15 terms on sac-6000.

Concurrency, on gp3 with a cold page cache and #1082 applied: a lookup of five popular terms took 1.7 ms instead of 4.4 ms on pubnet chunk 6465, and 1.9 ms instead of 4.6 ms on K7. Single-term lookups, and lookups on a volume already at its throughput limit, are unchanged.

roaring

go.mod requires v2.26.0 and replaces it with the head of RoaringBitmap/roaring#572, which sits on RoaringBitmap/roaring#566 and on roaring's master. The two make FastAnd and run-container intersections count or test before allocating, so a plan that matches nothing in a slab allocates only its empty result. #1087's slab walk runs that intersection on every slab. .golangci.yml allows this one replace.

roaring_contract_test.go tests roaring alone, so a version bump that breaks a property the event index relies on fails there first: FastAnd and the aggregations return containers that share no storage with their arguments, the value searches are read-only and inclusive, and an empty FastAnd allocates only its result. The last test covers every mix of array, bitmap and run terms up to four, and a sample up to seven. On v2.26.0 without the replace, it fails all 3,727 cases.

Known limitations

🤖 Generated with Claude Code

@tamirms
tamirms added this pull request to stack #1089 October 5, 2026 16:24
@tamirms
tamirms removed this pull request from stack #1089 October 8, 2026 08:18
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch from 1ac8a61 to 4f31522 Compare October 8, 2026 08:18
@tamirms
tamirms added this pull request to stack #1102 October 8, 2026 08:19
@tamirms tamirms changed the title events, deps: memory-map the cold MPHF, and roaring v2.26.0 events, deps: memory-map the cold MPHF and overlap cold reads, roaring v2.26.0 Oct 8, 2026
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch from 4f31522 to c8aac0c Compare October 8, 2026 08:34
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch 2 times, most recently from 44efd6d to 193bda0 Compare October 9, 2026 06:02
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch 2 times, most recently from 1d0ed30 to 3bdacff Compare October 15, 2026 19:15
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch 2 times, most recently from 0f5c469 to e66bb2c Compare October 21, 2026 09:12
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch 2 times, most recently from 22d71a7 to a434820 Compare October 21, 2026 09:52
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch from a434820 to a890554 Compare October 21, 2026 11:16
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch 4 times, most recently from e194ca3 to 1341976 Compare October 21, 2026 17:05
@tamirms
tamirms marked this pull request as ready for review October 21, 2026 17:12
@tamirms
tamirms requested a balanced review from Copilot October 21, 2026 17:12
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch from 1341976 to fb3ff5f Compare October 21, 2026 17:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The unreleased Roaring fork and mapped concurrent-read changes warrant human validation of compatibility and lifecycle safety.

No open issues 🎉

What changed in this PR

Improves cold-event query performance by memory-mapping indexes, batching term lookups, and overlapping packfile reads.

Changes:

  • Replaces whole-index reads with memory mapping and batched lookups.
  • Enables up to eight concurrent cold-event reads.
  • Updates Roaring dependencies with a temporary fork pin and adds bitmap contract tests.
File Description
go.sum Updates dependency checksums.
go.mod Updates Roaring and bitset; pins the temporary Roaring fork.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​roaring_contract_test.go Tests bitmap ownership, searches, concurrency, and allocations.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_reader.go Uses batched term lookups.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_reader_test.go Updates comments for mapped-index deletion behavior.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_index.go Documents batched lookup behavior.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_format.go Adds mapped index opening and batch lookup support.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_format_test.go Checks batch and individual lookup equivalence.
cmd/​stellar-rpc/​internal/​rpcv2/​query/​resolve.go Enables eight concurrent cold-event reads.

Review effort: Balanced

@tamirms
tamirms requested a review from marwen-abid October 21, 2026 17:35
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch from fb3ff5f to 879a0f4 Compare October 22, 2026 18:41
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch from 879a0f4 to 3e305ba Compare October 22, 2026 19:46
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch 3 times, most recently from b35752b to a42a881 Compare October 23, 2026 06:14
Base automatically changed from tamirms/events-packed-ledger-offsets to main October 23, 2026 07:08
tamirms and others added 2 commits October 23, 2026 08:08
…g v2.26.0

Each cold events read opened its chunk's index.hash by reading the whole
file into memory. It now memory-maps the file instead. The pages come
from the kernel's page cache, which every reader of the chunk shares, so
an open costs a map plus the pages its lookups touch, rather than a copy
of a file that grows with the chunk's term count.

LookupKeys now resolves all of a query's terms with one streamhash
QueryBatch call (stellar/streamhash#24) instead of one lookup per term.
On a cold page cache a lookup reads a few index.hash pages one after
another; QueryBatch requests every term's pages together, so a query's
lookups cost about one round of reads instead of one per term. On gp3
with the file evicted from the page cache, opening a 60M-term index and
looking up 15 terms took 2.5 ms instead of 17.3 ms at p50, and 5 terms
2.3 ms instead of 6.8 ms.

Cold events readers were opened with a concurrency of 0, so each lookup
and fetch issued its index.pack and events.pack reads one after another:
one per term for a lookup, and one per batch of a page's scattered
events for a fetch. The resolver now opens them with a concurrency of 8,
which packfile's ReadItems uses for both packs. Measured on gp3 with a
cold page cache, with index.pack's big terms split by slab (a later
change in this stack), lookups of several terms got 40 to 62% faster at
p50: five popular terms took 1.7 ms instead of 4.4 ms on pubnet chunk
6465, and 1.9 ms instead of 4.6 ms on a chunk folded to seven times its
density. Single-term lookups, and lookups on a volume already at its
throughput limit, are unchanged.

roaring moves from v2.18.2 to v2.26.0, which adds vectorized container
kernels. Its serialized form is roaring's portable format, so existing
artifacts read the same. go.mod replaces it with the head of
RoaringBitmap/roaring#572, which sits on RoaringBitmap/roaring#566,
until a release includes both: they count or test before allocating in
FastAnd and in run-container intersections, so an intersection that
comes out empty allocates nothing.

Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
Claude-Session: https://claude.ai/code/session_01JCzMKKrK3NstGKvaxsHhL2
Claude-Session: https://claude.ai/code/session_01Kkry5utNweWZqUKL88vJ2X
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new head of RoaringBitmap/roaring#572 looks up a one-key call's
key in every input, up to eight, before it allocates anything. So an
empty per-slab intersection allocates only its result bitmap, whatever
mix of array, bitmap and run containers its terms hold.

roaring_contract_test.go pins the roaring properties the event index
relies on, written against roaring alone so it runs first on any
version bump: FastAnd and the aggregations read their arguments and
return containers that share no storage with them, the value searches
are read-only and inclusive, and an empty FastAnd allocates only its
result. That last test makes the one-key call with every mix of array,
bitmap and run terms up to four terms and a sample of mixes up to
seven, on a whole and a partial slab, with the result emptied at each
term in turn. It fails on the previous pin, 64285f3, in 3,263 of its
3,727 cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wwsd8qTEGeSb7poFEYcgg9
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch from a42a881 to dd4d5dc Compare October 23, 2026 07:09

@marwen-abid marwen-abid left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Comment thread go.mod Resolved
@tamirms
tamirms merged commit dac748d into main Oct 24, 2026
18 checks passed
@tamirms
tamirms deleted the tamirms/roaring-2.26-mphf-mmap branch October 24, 2026 02:49
@tamirms tamirms moved this from To Do to Done in Platform Scrum Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants