Repository navigation
events: match queries one slab at a time - #1087
Merged
Merged
Conversation
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
from
October 8, 2026 08:18
76598f8 to
86fe919
Compare
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
3 times, most recently
from
October 9, 2026 06:02
2dc83cf to
ca3d216
Compare
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
2 times, most recently
from
October 15, 2026 19:15
60d5c4b to
baa1bd4
Compare
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
from
October 19, 2026 13:00
baa1bd4 to
af5b344
Compare
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
from
October 21, 2026 08:48
af5b344 to
13d4c27
Compare
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
3 times, most recently
from
October 21, 2026 09:52
9114839 to
6153b88
Compare
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
3 times, most recently
from
October 21, 2026 14:07
f183b8a to
02fe44f
Compare
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
2 times, most recently
from
October 21, 2026 17:05
84d7da5 to
edea772
Compare
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
from
October 21, 2026 17:15
edea772 to
2ab5c47
Compare
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
3 times, most recently
from
October 23, 2026 05:27
5891e7f to
cf8bc60
Compare
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
3 times, most recently
from
October 23, 2026 07:09
5353af6 to
4f9cc53
Compare
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
from
October 24, 2026 02:49
4f9cc53 to
d9ef98c
Compare
This was referenced Oct 26, 2026
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
from
October 28, 2026 08:05
d9ef98c to
70911f3
Compare
tamirms
marked this pull request as ready for review
October 28, 2026 08:37
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Batch reads continue after request cancellation, and the concurrency test can pass without exercising snapshot traversal during ingestion.
2 open issues
What changed in this PR
Replaces whole-chunk event bitmap matching with a shared, bidirectional slab walk, limiting candidate evaluation to relevant portions of the query window.
Changes:
- Expands filters into term plans and skips slabs that cannot contain candidates.
- Honors page-sized first fetches while splitting hot-store reads into bounded RocksDB batches.
- Adds oracle-based matching, slab-skipping, batching, and snapshot-safety coverage.
| File | Description |
|---|---|
cmd/stellar-rpc/internal/rpcv2/stores/event/slab_match.go |
Implements slab evaluation, directional traversal, and candidate streaming. |
cmd/stellar-rpc/internal/rpcv2/stores/event/slab_match_test.go |
Adds matching matrices, candidate-fetch checks, and slab-skipping tests. |
cmd/stellar-rpc/internal/rpcv2/stores/event/matches_differential_test.go |
Adds in-memory fixtures and concurrent-ingest snapshot coverage. |
cmd/stellar-rpc/internal/rpcv2/stores/event/match.go |
Introduces term plans, integrates slab matching, and honors first-batch hints. |
cmd/stellar-rpc/internal/rpcv2/stores/event/match_test.go |
Updates matching coverage and adds plan-width and batch-size checks. |
cmd/stellar-rpc/internal/rpcv2/stores/event/hot_store.go |
Splits event fetches into bounded RocksDB reads. |
cmd/stellar-rpc/internal/rpcv2/stores/event/hot_store_test.go |
Extends large-batch coverage across multiple read batches. |
cmd/stellar-rpc/internal/rpcv2/stores/event/concurrent_bitmaps.go |
Clarifies read-only bitmap and snapshot-lifetime documentation. |
Review effort: Balanced
Shaptic
approved these changes
Nov 4, 2026
Shaptic
left a comment
Contributor
There was a problem hiding this comment.
Mostly minor notes, can be merged without resolving any of them, but wanted to throw some thoughts out
A getEvents query was answered by intersecting each filter's term bitmaps across the whole chunk, unioning the results across filters, and only then restricting to the window. Its cost tracked the terms' size across the chunk, not the window the client asked for or the page it would receive. The engine now walks the window one slab at a time: 65,536 consecutive event ids, roaring's container size, in the requested direction. Each filter becomes plans that are plain ANDs of terms. On each slab, every plan is one FastAnd over the slab's ids in the window and the plan's terms, and the plans' results are ORed and handed to the fetch. Before opening a slab, the walk asks each term for its next id and jumps to the first slab that can hold a candidate, so a rare term opens only the slabs it appears in. Work scales with the window and the page, memory per query with one container, and both directions share one path. The first batch is the caller's page size, so a full page arrives in one fetch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
A page now arrives in one fetch, so the hot store passed up to 10,001 ids to one BatchMultiGet. RocksDB pins every block that call touches until it returns, and a sparse v1 page could pin about 313 MiB of block cache. FetchEvents now calls BatchMultiGet once per matchBatchSize ids, which bounds the pin to about 16 MiB. Event rows never change once written, so the slices need no shared snapshot. The cold store keeps one read per page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wwsd8qTEGeSb7poFEYcgg9
slabPlan.eval built its FastAnd argument slice with make on every call, which goes to the heap for plans of four or more terms: at least 48 bytes per plan per slab, about 2 MB per page at 256 filters. The stepper now sizes one buffer for its widest plan and eval reuses it, so an empty slab allocates only FastAnd's result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wwsd8qTEGeSb7poFEYcgg9
A plan is at most the contract id, the event type, four topics and one topic-count bucket, so with the slab FastAnd gets at most eight inputs, the width the pinned roaring checks before it allocates. The test walks every filter shape and topic-count constraint and requires that bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wwsd8qTEGeSb7poFEYcgg9
A page now arrives in one fetch, so one hot FetchEvents can make about 20 BatchMultiGet calls, and a canceled request still made all of them. FetchEvents now checks ctx before each call, so a cancel costs at most the call in flight, as it did when the matcher fetched 512 ids at a time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
from
November 4, 2026 09:50
218187d to
c9ed941
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What
Rewrites how the events store matches a getEvents query. It now walks the query's window one slab at a time (65,536 consecutive event ids, roaring's container size) instead of combining whole-chunk bitmaps before the first fetch.
Tests:
TestMatches_FetchesOnlyTrueCandidatesrequires the fetched ids to be exactly the query's matches; the matrices inslab_match_test.gocompare against an oracle computed without the index;TestSlabStepperSkipsCandidateFreeSlabspins which slabs the walk opens;TestTermPlansAtMostSevenTermspins that no plan has more than seven terms, soFastAndgets at most the eight inputs roaring checks before it allocates;roaring_contract_test.go(from #1086) pins, against roaring alone and under-race, the properties the shared index bitmaps rely on, including that aFastAndof the walk's shape that comes out empty allocates only its result, for every mix of array, bitmap and run terms;TestMatches_ConcurrentIngestBorrowSafetyruns walks while ingest keeps writing.Why
A query is a list of filters, each an AND of terms (contract id, topics, event type), and the candidates are the ids that satisfy any filter inside the window. Today the engine intersects each filter's term bitmaps across the whole chunk, unions the results, and only then restricts to the window, all before the first event is fetched. Its cost tracks the terms' size across the chunk rather than the window or the page: a page of 1,000 events from a popular contract with a topic constraint first intersects that contract's millions of ids.
This PR turns each filter into plans, each a plain AND of terms (a topic-count range becomes one plan per count), and walks the window slab by slab in the requested direction. On each slab, every plan is one
FastAndover the slab's ids in the window and the plan's terms, and the plans' results are ORed and handed to the fetch. Before opening a slab, the walk asks each term for its next id (NextValue/PreviousValue) and jumps to the first slab that can hold a candidate, so a rare term opens only the slabs where it has events. The matching work scales with the window and the page, the walk holds one slab's results at a time, and both directions share one path. Every plan'sFastAndreuses one argument slice, so with the roaring pinned in #1086 a plan that matches nothing in a slab allocates only its empty result. The first batch is the caller's page size, so a full page arrives in one fetch. The hot store reads that fetch from RocksDB 512 ids at a time, because RocksDB pins every block a call touches until the call returns: one call for a sparse page of 10,000 events could pin about 313 MiB of block cache, and 512 ids pin about 16 MiB.Known limitations
🤖 Generated with Claude Code