Skip to content

events: keep 8 reads in flight per cold events lookup or fetch - #1083

Closed
tamirms wants to merge 12 commits into
tamirms/events-split-termsfrom
tamirms/events-cold-read-concurrency
Closed

tamirms wants to merge 12 commits into
tamirms/events-split-termsfrom
tamirms/events-cold-read-concurrency

Conversation

@tamirms

@tamirms tamirms commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Cold events readers were opened with a concurrency of 0, so each lookup
and fetch issued its reads one after another: one per term for a
lookup, and one per batch of a page's scattered events for a fetch. The
resolver now opens them with a concurrency of 8, which packfile's
ReadItems uses for index.pack and events.pack alike.

Measured on gp3 with a cold page cache, lookups of several terms got 40
to 62% faster at p50: five popular terms took 1.7 ms instead of 4.4 ms
on pubnet chunk 6465, and 1.9 ms instead of 4.6 ms on a chunk folded to
seven times its density. Single-term lookups, and lookups on a volume
already at its throughput limit, are unchanged.

🤖 Generated with Claude Code

tamirms and others added 3 commits October 5, 2026 11:07
Open reads the last 256 KiB of every pack. When the index, app data and
trailer are larger, it reads the whole index region again, up to 256 KiB
more than it needs; events.pack's index reaches 0.8 to 1 MB on large
chunks. For a ledger pack 256 KiB is too much: its tail is under 42 KB
(16 to 28 KB on real chunks), and a request opens every chunk it routes
to.

ReaderOptions.FirstRead now sizes Open's first read from the file size,
so a store whose tail grows with the file can follow it; nil keeps
256 KiB. When the first read falls short of the tail, Open reads only
the missing bytes in front of it. The ledger cold reader sets 64 KiB,
which always covers its tail.

doOpen now hands the opened file to openFile, so a test can record every
read Open makes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
The reader formed its reads by two rules at once. Adjacent records
merged into one ReadAt of up to 1 MiB, and ReadItems also capped every
read at ceil(positions / Concurrency) positions. With Concurrency above
1 the cap split runs of adjacent records: in a replay of pubnet queries
with 8 reads in flight, one term's 156 adjacent index records went out
as 6 to 8 small reads, and whole-chunk windows took 23 ms at p50 instead
of 13 ms. The 1 MiB bound let a read grow past one EBS I/O: a cold 1 MiB
read takes 10.9 ms on gp3 against 1.3 ms for 256 KiB.

Adjacent records now merge by size alone into reads of up to 256 KiB,
and a larger record is read alone. Concurrency changes how many of
ReadItems' reads are in flight, not which reads it makes. ReadRange
applies the same rule, one read at a time. Every read goes into the
record workspace's scratch buffer, so the pool of 1 MiB read buffers and
the one-off allocations for larger records are gone.

Two older ReadItems tests are deleted; the new read-log tests cover
their cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
Open decoded the whole offset index into an []int64 of record offsets:
8 bytes per record held for the Reader's life and one FOR decode per
record on every open, though a request touches a few records and every
request opens its packs. With the file in the page cache, an open
through its first read and Close took 95 µs and 427 KB at 12,600
records (an index.pack), 326 µs and 1.24 MB at 80,000 (a small chunk's
events.pack) and 1.2 ms and 5.7 MB at 450,000 (a large chunk's).

The index section is now [groups][directory][CRC32C]. Each group of 128
records keeps one FOR group of their sizes, and a 17-byte directory
entry per group holds the file offset of its first record, where its
sizes end, the position of its first item and a flags byte, which no
flag uses yet. Open checks the CRC and the directory, which keeps each
group's records inside the records region on its own, and decodes no
group. A read decodes the groups it touches into its pooled workspace
and checks each one; a group that fails fails only the reads that touch
it, with ErrCorrupt. A lookup finds the group and the record by binary
search over first items, so it does not assume equal records. The
trailer version goes to 2.

The same opens now take 51, 60 and 82 µs and allocate 267, 288 and
402 KB. On BenchmarkReader's 200-record packs, a single ReadItem pays
one group decode, about 0.4 µs; ReadRange stays within 1% of before and
ReadItems within 3.5%. On a 450,000-record pack, a lookup's two binary
searches take about 50 ns.

Open keeps the index and app data in one buffer of the tail's size, the
refill buffer or a copy of the tail, so a large first read is not kept
for a small tail. The offsets table goes, with the record-count guard
that protected its allocation, itemsInRecord and its panics, and the
file-level record-count check, which the per-group first-item checks
replace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
tamirms and others added 9 commits October 5, 2026 17:04
Every request opens its packs, and each open reads the end of the file
into a new buffer of FirstRead bytes: 256 KiB by default and for
events.pack, 64 KiB for ledger packs. The Reader keeps a copy of the
tail or the second read's buffer, never the first read's, so that
buffer is garbage once the open finishes. It is most of what an open
allocates: 262 of the 267 KB that BenchmarkOpen's Open, TotalItems and
Close allocated at 12,600 records, with the file in the page cache.

The first read now goes into the scratch buffer of a workspace from
recordWorkspacePool, which already holds every read's buffer, and the
workspace goes back to the pool when the open finishes, whether or not
it succeeded. A scratch buffer smaller than the first read is replaced
by one of the right size, so the pool's buffers settle at the largest
read the process makes. At 12,600, 80,000 and 450,000 records the
benchmark now takes 35, 43 and 74 µs instead of 55, 67 and 94 (medians
of 10 interleaved runs), and allocates 5.5, 28 and 143 KB instead of
267, 288 and 402.

The pool needs no cap: the buffer's size is FirstRead(fileSize), which
the store sets and the file size bounds, so no file's contents can make
the pool keep a large buffer. What an open still allocates is mostly the
tail the Reader keeps, which is not pooled because AppData returns a
view into it that a caller may hold past Close.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
events.pack's app data holds a chunk's LedgerOffsets as 10,000
big-endian uint32 cumulative event counts: 40,009 bytes that every
cold-reader open copies out of its tail read, checksums and decodes.
A ledger's event count needs about 11 bits on pubnet, and an eventless
chunk, which is every pre-Soroban chunk, pays the same 40,009 bytes for
zeros.

Version 2 keeps the 9-byte header and stores each ledger's event count
with intpack.EncodeGroup, in groups of 128 ledgers, which the decoder
walks back from the end of the blob. A real pubnet chunk's blob shrinks
to 13,780 bytes and an eventless chunk's to 1,654. Decoding a full
chunk's blob, 10,000 ledgers of 900 to 1,200 events, takes 19 µs instead
of 29, with the same two allocations.

The decoder bounds the ledger count by chunk.LedgersPerChunk before it
allocates, requires the groups to cover the payload exactly, and rejects
counts whose sum overflows uint32, the only way the cumulative array
could decrease. The decoder, its error and the version constant become
unexported; nothing outside the package used them. A pack written with
version 1 fails to open with an unsupported-version error and has to be
rebuilt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
Each cold events read opened its chunk's index.hash by reading the whole
file into memory. It now memory-maps the file instead. The pages come
from the kernel's page cache, which every reader of the chunk shares, so
an open costs a map plus the pages its lookups touch, rather than a copy
of a file that grows with the chunk's term count.

roaring moves from v2.18.2 to v2.26.0, which adds vectorized container
kernels. Its serialized form is roaring's portable format, so existing
artifacts read the same.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
A getEvents query was answered by intersecting each filter's term bitmaps
across the whole chunk, unioning the results across filters, and only then
restricting to the window. Its cost tracked the terms' size across the
chunk, not the window the client asked for or the page it would receive.

The engine now walks the window one slab at a time: 65,536 consecutive
event ids, roaring's container size, in the requested direction. Each
filter becomes plans that are plain ANDs of terms. On each slab, every plan
is one FastAnd over the slab's ids in the window and the plan's terms, and
the plans' results are ORed and handed to the fetch. Before opening a slab,
the walk asks each term for its next id and jumps to the first slab that
can hold a candidate, so a rare term opens only the slabs it appears in.
Work scales with the window and the page, memory per query with one
container, and both directions share one path. The first batch is the
caller's page size, so a full page arrives in one fetch.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
LookupKeys now takes the range of event ids the query needs and reports
the range its answer covers. Matches first asks for the leading four slabs
of the window, in the walk's direction. It asks for the rest only if the
page is still not full and the first answer did not already cover it.

Both readers still return whole terms, so they report the whole id space
and every query runs in one stage, as before. The point is the next step:
a reader that stores a large term in pieces can then read only the pieces
a page reaches.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
A record closed only on its item count, so a few large items made it
large, and a read of any item in it reads all of it. index.pack's large
bitmaps share 128-item records with small ones, so a lookup of a rare
term reads whatever its neighbors weigh, and events.pack's records grow
when other contracts emit big events. index.pack's planned byte-only
budget needs records limited by bytes alone.

WriterOptions.MaxRecordBytes closes a record before an item that would
take its raw bytes past the limit, so only a record of one item can
exceed it. ItemsPerRecord 0 now means no item limit and needs
MaxRecordBytes. The default of 128 for an unset ItemsPerRecord goes, and
Create rejects options with no limit, and a byte limit with one item per
record. Every production writer already sets ItemsPerRecord.

A record closed early holds fewer items than the limit, so its index
group gets a count column, a FOR group of its records' item counts
marked by the directory's flag bit, and the directory's first items
become running totals. Every group has counts when there is no item
limit, and a pack of one item per record never does. Open checks each
group's flag and item span, and a read checks a group's counts when it
decodes it. Lookups already search first items, so they need no change.
A record keeps the shape the pack's limit sets, so one closed at one
item still carries its item size index and CRC.

A pack whose records all fill is byte-identical to before, and a reader
without counts rejects a pack with them, so the trailer version stays
at 2. Verify ends a content-hash chunk where each record ends, as the
writer does. The trailer check for an item limit above 0 goes with its
test, as do Open's reading of 0 as 1, the Reader's own copies of the
item count and the item limit, and the test helper writeTestPackfile,
which read an unset ItemsPerRecord as 1.

Over six interleaved rounds with the file in the page cache,
BenchmarkReader stays within 2.4% of before and BenchmarkOpen within
2.3%. On the same records closed at 128 items by a byte limit under an
item limit of 256, so every group has counts, a ReadItem costs 0.3 µs
more for the count column's decode, and ReadItems and ReadRange under 1%
more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
An events.pack record closes only after 128 payloads, so a few large
events make it large, and a read of any event in it fetches and decodes
all of it. One contract's big events thus inflate reads of other
contracts' events.

The events.pack writer now sets MaxRecordBytes to 128 KiB. Payloads
average about 250 bytes, so 128 of them come to about 32 KB and real
records still close at 128 items: production packs are unchanged, and
their index stores no item counts. The reader addresses payloads by
position and needs no change. The getEvents design doc now describes
both record limits and finds an event's record through the offset
index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
…slabs

A popular term's bitmap is read whole for every page that touches it: on
pubnet chunk 6410 the fifteen most popular terms are 250 KiB to 1.08 MiB
each, so a popular page reads about 9 MB of index to answer from a few
slabs of it. And index.pack's 128-entry records make its tail follow the
term count, which no fixed first read covers on big chunks, so every open
there pays a second round trip.

A term whose serialized bitmap is over 64 KiB is now stored as one entry
per slab of the chunk, zero-length where it has no events, and LookupKeys
reads only the slabs the window reaches and reports their span as
covered. The app data names the split terms after the chunk's slab
count, one 8-byte row of slot and fingerprint each; the stamp moves to
0x02 and the blob has an exact length. Records close at 16 KiB with no
entry limit, so the tail follows the file size, and Open's first read is
max(64 KiB, 4 bytes per 16 KiB of file), rounded up to 4 KiB.

Measured on gp3 with a cold page cache, against the code before this
change: five popular terms over four slabs took 1.8 ms p50 instead of
11.9 ms on a pubnet chunk folded to seven times its density, and 12 ms
instead of 118 ms with eight query streams each reading its own copy. A
rare term took 3.0 ms instead of 5.3 ms on sac-6000, whose tail now fits
one read. A read of a split term's whole chunk rebuilds the term from
its slab entries, about 0.5 ms more CPU per term on the folded chunk
than decoding one whole bitmap.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCzMKKrK3NstGKvaxsHhL2
Cold events readers were opened with a concurrency of 0, so each lookup
and fetch issued its reads one after another: one per term for a
lookup, and one per batch of a page's scattered events for a fetch. The
resolver now opens them with a concurrency of 8, which packfile's
ReadItems uses for index.pack and events.pack alike.

Measured on gp3 with a cold page cache, lookups of several terms got 40
to 62% faster at p50: five popular terms took 1.7 ms instead of 4.4 ms
on pubnet chunk 6465, and 1.9 ms instead of 4.6 ms on a chunk folded to
seven times its density. Single-term lookups, and lookups on a volume
already at its throughput limit, are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCzMKKrK3NstGKvaxsHhL2
@tamirms
tamirms removed this pull request from stack #1081 October 5, 2026 16:19
@tamirms
tamirms force-pushed the tamirms/events-split-terms branch from 9c001bd to b7c7396 Compare October 5, 2026 16:19
@tamirms
tamirms force-pushed the tamirms/events-cold-read-concurrency branch from 1ba5ac5 to 462b735 Compare October 5, 2026 16:19
@tamirms
tamirms added this pull request to stack #1089 October 5, 2026 16:24
@tamirms
tamirms removed this pull request from stack #1089 October 8, 2026 08:18
@tamirms
tamirms force-pushed the tamirms/events-split-terms branch from b7c7396 to 9dd7c16 Compare October 8, 2026 08:18
@tamirms

tamirms commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Squashed into #1086.

@tamirms tamirms closed this Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant