Skip to content

events: match queries one slab at a time - #1041

Merged
tamirms merged 1 commit into
tamirms/events-pack-byte-limitfrom
tamirms/events-slab-matcher
Oct 5, 2026
Merged

tamirms merged 1 commit into
tamirms/events-pack-byte-limitfrom
tamirms/events-slab-matcher

Conversation

@tamirms

@tamirms tamirms commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

What

Rewrites how the events store matches a getEvents query. It now walks the query's window one slab at a time (65,536 consecutive event ids, roaring's container size) instead of combining whole-chunk bitmaps before the first fetch.

Tests: TestMatches_FetchesOnlyTrueCandidates requires the fetched ids to be exactly the query's matches; the matrices in slab_match_test.go compare against an oracle computed without the index; TestSlabStepperSkipsCandidateFreeSlabs pins which slabs the walk opens; roaring_contract_test.go pins, against roaring alone and under -race, the properties the shared index bitmaps rely on; TestMatches_ConcurrentIngestBorrowSafety holds index snapshots across a walk while ingest keeps writing.

Why

A query is a list of filters, each an AND of terms (contract id, topics, event type), and the candidates are the ids that satisfy any filter inside the window. Today the engine intersects each filter's term bitmaps across the whole chunk, unions the results, and only then restricts to the window, all before the first event is fetched. Its cost tracks the terms' size across the chunk rather than the window or the page: a page of 1,000 events from a popular contract with a topic constraint first intersects that contract's millions of ids.

This PR turns each filter into plans, each a plain AND of terms (a topic-count range becomes one plan per count), and walks the window slab by slab in the requested direction. On each slab, every plan is one FastAnd over the slab's ids in the window and the plan's terms, and the plans' results are ORed and handed to the fetch. Before opening a slab, the walk asks each term for its next id (NextValue / PreviousValue) and jumps to the first slab that can hold a candidate, so a rare term opens only the slabs where it has events. Work scales with the window and the page, memory per query with one container, and both directions share one path. The first batch is the caller's page size, so a full page arrives in one fetch.

Known limitations

🤖 Generated with Claude Code

@tamirms
tamirms added this pull request to stack #1044 September 25, 2026 19:41
@tamirms
tamirms force-pushed the tamirms/events-slab-matcher branch from 847cf12 to 809e74f Compare September 25, 2026 20:35
@tamirms
tamirms force-pushed the tamirms/events-slab-matcher branch from 809e74f to 968c1a9 Compare October 5, 2026 10:53
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch from 8816035 to a85b032 Compare October 5, 2026 10:53
@tamirms
tamirms removed this pull request from stack #1044 October 5, 2026 10:56
@tamirms
tamirms added this pull request to stack #1081 October 5, 2026 10:56
A getEvents query was answered by intersecting each filter's term bitmaps
across the whole chunk, unioning the results across filters, and only then
restricting to the window. Its cost tracked the terms' size across the
chunk, not the window the client asked for or the page it would receive.

The engine now walks the window one slab at a time: 65,536 consecutive
event ids, roaring's container size, in the requested direction. Each
filter becomes plans that are plain ANDs of terms. On each slab, every plan
is one FastAnd over the slab's ids in the window and the plan's terms, and
the plans' results are ORed and handed to the fetch. Before opening a slab,
the walk asks each term for its next id and jumps to the first slab that
can hold a candidate, so a rare term opens only the slabs it appears in.
Work scales with the window and the page, memory per query with one
container, and both directions share one path. The first batch is the
caller's page size, so a full page arrives in one fetch.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
@tamirms
tamirms removed this pull request from stack #1081 October 5, 2026 16:19
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch from a85b032 to 1ac8a61 Compare October 5, 2026 16:19
@tamirms
tamirms force-pushed the tamirms/events-slab-matcher branch from 968c1a9 to 76598f8 Compare October 5, 2026 16:19
Base automatically changed from tamirms/roaring-2.26-mphf-mmap to tamirms/events-pack-byte-limit October 5, 2026 16:19
@tamirms
tamirms merged commit 76598f8 into tamirms/events-pack-byte-limit Oct 5, 2026
14 checks passed
@tamirms
tamirms deleted the tamirms/events-slab-matcher branch October 5, 2026 16:19
@tamirms
tamirms restored the tamirms/events-slab-matcher branch October 5, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant