Skip to content

events, deps: memory-map the cold MPHF, and roaring v2.26.0 - #1040

Merged
tamirms merged 1 commit into
tamirms/events-pack-byte-limitfrom
tamirms/roaring-2.26-mphf-mmap
Oct 5, 2026
Merged

tamirms merged 1 commit into
tamirms/events-pack-byte-limitfrom
tamirms/roaring-2.26-mphf-mmap

Conversation

@tamirms

@tamirms tamirms commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

What

  • The cold events reader memory-maps each chunk's index.hash (streamhash.Open) instead of reading the whole file into memory (os.ReadFile + streamhash.OpenBytes).
  • roaring moves from v2.18.2 to v2.26.0.

Why

A cold events read opens its chunk's index for every request. Reading it whole copies a file whose size grows with the chunk's term count, about 18.5 MB per request at high term cardinality. Mapped, the pages come from the kernel's page cache, which every reader of the chunk shares, so a request pays only for the pages its lookups touch.

roaring v2.26.0 adds vectorized container kernels and drops a clone from its bulk AND and OR, which the match engine in the next PR runs on every slab. Its serialized form is roaring's portable format, so existing artifacts read the same.

This is the first of four PRs split out of #968 and tamirms/cold-index-parts, stacked on #1034.

Known limitations

N/A

🤖 Generated with Claude Code

@socket-security

socket-security Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Updatedgolang/​github.com/​RoaringBitmap/​roaring/​v2@​v2.18.2 ⏵ v2.26.099 +1100100100100

View full report

@tamirms
tamirms added this pull request to stack #1044 September 25, 2026 19:41
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch from ac4db2a to 8816035 Compare September 25, 2026 20:35
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch from 8816035 to a85b032 Compare October 5, 2026 10:53
@tamirms
tamirms force-pushed the tamirms/routed-fingerprint branch from 27ec7c2 to b828b0d Compare October 5, 2026 10:53
@tamirms
tamirms removed this pull request from stack #1044 October 5, 2026 10:56
@tamirms
tamirms changed the base branch from tamirms/routed-fingerprint to tamirms/events-packed-ledger-offsets October 5, 2026 10:56
@tamirms
tamirms added this pull request to stack #1081 October 5, 2026 10:56
Each cold events read opened its chunk's index.hash by reading the whole
file into memory. It now memory-maps the file instead. The pages come
from the kernel's page cache, which every reader of the chunk shares, so
an open costs a map plus the pages its lookups touch, rather than a copy
of a file that grows with the chunk's term count.

roaring moves from v2.18.2 to v2.26.0, which adds vectorized container
kernels. Its serialized form is roaring's portable format, so existing
artifacts read the same.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
@tamirms
tamirms removed this pull request from stack #1081 October 5, 2026 16:19
@tamirms
tamirms force-pushed the tamirms/events-packed-ledger-offsets branch from be176d8 to 4f3c289 Compare October 5, 2026 16:19
@tamirms
tamirms force-pushed the tamirms/roaring-2.26-mphf-mmap branch from a85b032 to 1ac8a61 Compare October 5, 2026 16:19
Base automatically changed from tamirms/events-packed-ledger-offsets to tamirms/events-pack-byte-limit October 5, 2026 16:19
@tamirms
tamirms merged commit 1ac8a61 into tamirms/events-pack-byte-limit Oct 5, 2026
13 of 14 checks passed
@tamirms
tamirms deleted the tamirms/roaring-2.26-mphf-mmap branch October 5, 2026 16:19
@tamirms
tamirms restored the tamirms/roaring-2.26-mphf-mmap branch October 5, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant