Repository navigation
Conversation
Open reads the last 256 KiB of every pack. When the index, app data and trailer are larger, it reads the whole index region again, up to 256 KiB more than it needs; events.pack's index reaches 0.8 to 1 MB on large chunks. For a ledger pack 256 KiB is too much: its tail is under 42 KB (16 to 28 KB on real chunks), and a request opens every chunk it routes to. ReaderOptions.FirstRead now sizes Open's first read from the file size, so a store whose tail grows with the file can follow it; nil keeps 256 KiB. When the first read falls short of the tail, Open reads only the missing bytes in front of it. The ledger cold reader sets 64 KiB, which always covers its tail. doOpen now hands the opened file to openFile, so a test can record every read Open makes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
The reader formed its reads by two rules at once. Adjacent records merged into one ReadAt of up to 1 MiB, and ReadItems also capped every read at ceil(positions / Concurrency) positions. With Concurrency above 1 the cap split runs of adjacent records: in a replay of pubnet queries with 8 reads in flight, one term's 156 adjacent index records went out as 6 to 8 small reads, and whole-chunk windows took 23 ms at p50 instead of 13 ms. The 1 MiB bound let a read grow past one EBS I/O: a cold 1 MiB read takes 10.9 ms on gp3 against 1.3 ms for 256 KiB. Adjacent records now merge by size alone into reads of up to 256 KiB, and a larger record is read alone. Concurrency changes how many of ReadItems' reads are in flight, not which reads it makes. ReadRange applies the same rule, one read at a time. Every read goes into the record workspace's scratch buffer, so the pool of 1 MiB read buffers and the one-off allocations for larger records are gone. Two older ReadItems tests are deleted; the new read-log tests cover their cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
Open decoded the whole offset index into an []int64 of record offsets: 8 bytes per record held for the Reader's life and one FOR decode per record on every open, though a request touches a few records and every request opens its packs. With the file in the page cache, an open through its first read and Close took 95 µs and 427 KB at 12,600 records (an index.pack), 326 µs and 1.24 MB at 80,000 (a small chunk's events.pack) and 1.2 ms and 5.7 MB at 450,000 (a large chunk's). The index section is now [groups][directory][CRC32C]. Each group of 128 records keeps one FOR group of their sizes, and a 17-byte directory entry per group holds the file offset of its first record, where its sizes end, the position of its first item and a flags byte, which no flag uses yet. Open checks the CRC and the directory, which keeps each group's records inside the records region on its own, and decodes no group. A read decodes the groups it touches into its pooled workspace and checks each one; a group that fails fails only the reads that touch it, with ErrCorrupt. A lookup finds the group and the record by binary search over first items, so it does not assume equal records. The trailer version goes to 2. The same opens now take 51, 60 and 82 µs and allocate 267, 288 and 402 KB. On BenchmarkReader's 200-record packs, a single ReadItem pays one group decode, about 0.4 µs; ReadRange stays within 1% of before and ReadItems within 3.5%. On a 450,000-record pack, a lookup's two binary searches take about 50 ns. Open keeps the index and app data in one buffer of the tail's size, the refill buffer or a copy of the tail, so a large first read is not kept for a small tail. The offsets table goes, with the record-count guard that protected its allocation, itemsInRecord and its panics, and the file-level record-count check, which the per-group first-item checks replace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
Every request opens its packs, and each open reads the end of the file into a new buffer of FirstRead bytes: 256 KiB by default and for events.pack, 64 KiB for ledger packs. The Reader keeps a copy of the tail or the second read's buffer, never the first read's, so that buffer is garbage once the open finishes. It is most of what an open allocates: 262 of the 267 KB that BenchmarkOpen's Open, TotalItems and Close allocated at 12,600 records, with the file in the page cache. The first read now goes into the scratch buffer of a workspace from recordWorkspacePool, which already holds every read's buffer, and the workspace goes back to the pool when the open finishes, whether or not it succeeded. A scratch buffer smaller than the first read is replaced by one of the right size, so the pool's buffers settle at the largest read the process makes. At 12,600, 80,000 and 450,000 records the benchmark now takes 35, 43 and 74 µs instead of 55, 67 and 94 (medians of 10 interleaved runs), and allocates 5.5, 28 and 143 KB instead of 267, 288 and 402. The pool needs no cap: the buffer's size is FirstRead(fileSize), which the store sets and the file size bounds, so no file's contents can make the pool keep a large buffer. What an open still allocates is mostly the tail the Reader keeps, which is not pooled because AppData returns a view into it that a caller may hold past Close. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
events.pack's app data holds a chunk's LedgerOffsets as 10,000 big-endian uint32 cumulative event counts: 40,009 bytes that every cold-reader open copies out of its tail read, checksums and decodes. A ledger's event count needs about 11 bits on pubnet, and an eventless chunk, which is every pre-Soroban chunk, pays the same 40,009 bytes for zeros. Version 2 keeps the 9-byte header and stores each ledger's event count with intpack.EncodeGroup, in groups of 128 ledgers, which the decoder walks back from the end of the blob. A real pubnet chunk's blob shrinks to 13,780 bytes and an eventless chunk's to 1,654. Decoding a full chunk's blob, 10,000 ledgers of 900 to 1,200 events, takes 19 µs instead of 29, with the same two allocations. The decoder bounds the ledger count by chunk.LedgersPerChunk before it allocates, requires the groups to cover the payload exactly, and rejects counts whose sum overflows uint32, the only way the cumulative array could decrease. The decoder, its error and the version constant become unexported; nothing outside the package used them. A pack written with version 1 fails to open with an unsupported-version error and has to be rebuilt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
Each cold events read opened its chunk's index.hash by reading the whole file into memory. It now memory-maps the file instead. The pages come from the kernel's page cache, which every reader of the chunk shares, so an open costs a map plus the pages its lookups touch, rather than a copy of a file that grows with the chunk's term count. roaring moves from v2.18.2 to v2.26.0, which adds vectorized container kernels. Its serialized form is roaring's portable format, so existing artifacts read the same. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
A getEvents query was answered by intersecting each filter's term bitmaps across the whole chunk, unioning the results across filters, and only then restricting to the window. Its cost tracked the terms' size across the chunk, not the window the client asked for or the page it would receive. The engine now walks the window one slab at a time: 65,536 consecutive event ids, roaring's container size, in the requested direction. Each filter becomes plans that are plain ANDs of terms. On each slab, every plan is one FastAnd over the slab's ids in the window and the plan's terms, and the plans' results are ORed and handed to the fetch. Before opening a slab, the walk asks each term for its next id and jumps to the first slab that can hold a candidate, so a rare term opens only the slabs it appears in. Work scales with the window and the page, memory per query with one container, and both directions share one path. The first batch is the caller's page size, so a full page arrives in one fetch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
LookupKeys now takes the range of event ids the query needs and reports the range its answer covers. Matches first asks for the leading four slabs of the window, in the walk's direction. It asks for the rest only if the page is still not full and the first answer did not already cover it. Both readers still return whole terms, so they report the whole id space and every query runs in one stage, as before. The point is the next step: a reader that stores a large term in pieces can then read only the pieces a page reaches. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
A record closed only on its item count, so a few large items made it large, and a read of any item in it reads all of it. index.pack's large bitmaps share 128-item records with small ones, so a lookup of a rare term reads whatever its neighbors weigh, and events.pack's records grow when other contracts emit big events. index.pack's planned byte-only budget needs records limited by bytes alone. WriterOptions.MaxRecordBytes closes a record before an item that would take its raw bytes past the limit, so only a record of one item can exceed it. ItemsPerRecord 0 now means no item limit and needs MaxRecordBytes. The default of 128 for an unset ItemsPerRecord goes, and Create rejects options with no limit, and a byte limit with one item per record. Every production writer already sets ItemsPerRecord. A record closed early holds fewer items than the limit, so its index group gets a count column, a FOR group of its records' item counts marked by the directory's flag bit, and the directory's first items become running totals. Every group has counts when there is no item limit, and a pack of one item per record never does. Open checks each group's flag and item span, and a read checks a group's counts when it decodes it. Lookups already search first items, so they need no change. A record keeps the shape the pack's limit sets, so one closed at one item still carries its item size index and CRC. A pack whose records all fill is byte-identical to before, and a reader without counts rejects a pack with them, so the trailer version stays at 2. Verify ends a content-hash chunk where each record ends, as the writer does. The trailer check for an item limit above 0 goes with its test, as do Open's reading of 0 as 1, the Reader's own copies of the item count and the item limit, and the test helper writeTestPackfile, which read an unset ItemsPerRecord as 1. Over six interleaved rounds with the file in the page cache, BenchmarkReader stays within 2.4% of before and BenchmarkOpen within 2.3%. On the same records closed at 128 items by a byte limit under an item limit of 256, so every group has counts, a ReadItem costs 0.3 µs more for the count column's decode, and ReadItems and ReadRange under 1% more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
An events.pack record closes only after 128 payloads, so a few large events make it large, and a read of any event in it fetches and decodes all of it. One contract's big events thus inflate reads of other contracts' events. The events.pack writer now sets MaxRecordBytes to 128 KiB. Payloads average about 250 bytes, so 128 of them come to about 32 KB and real records still close at 128 items: production packs are unchanged, and their index stores no item counts. The reader addresses payloads by position and needs no change. The getEvents design doc now describes both record limits and finds an event's record through the offset index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
…slabs A popular term's bitmap is read whole for every page that touches it: on pubnet chunk 6410 the fifteen most popular terms are 250 KiB to 1.08 MiB each, so a popular page reads about 9 MB of index to answer from a few slabs of it. And index.pack's 128-entry records make its tail follow the term count, which no fixed first read covers on big chunks, so every open there pays a second round trip. A term whose serialized bitmap is over 64 KiB is now stored as one entry per slab of the chunk, zero-length where it has no events, and LookupKeys reads only the slabs the window reaches and reports their span as covered. The app data names the split terms after the chunk's slab count, one 8-byte row of slot and fingerprint each; the stamp moves to 0x02 and the blob has an exact length. Records close at 16 KiB with no entry limit, so the tail follows the file size, and Open's first read is max(64 KiB, 4 bytes per 16 KiB of file), rounded up to 4 KiB. Measured on gp3 with a cold page cache, against the code before this change: five popular terms over four slabs took 1.8 ms p50 instead of 11.9 ms on a pubnet chunk folded to seven times its density, and 12 ms instead of 118 ms with eight query streams each reading its own copy. A rare term took 3.0 ms instead of 5.3 ms on sac-6000, whose tail now fits one read. A read of a split term's whole chunk rebuilds the term from its slab entries, about 0.5 ms more CPU per term on the folded chunk than decoding one whole bitmap. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCzMKKrK3NstGKvaxsHhL2
Cold events readers were opened with a concurrency of 0, so each lookup and fetch issued its reads one after another: one per term for a lookup, and one per batch of a page's scattered events for a fetch. The resolver now opens them with a concurrency of 8, which packfile's ReadItems uses for index.pack and events.pack alike. Measured on gp3 with a cold page cache, lookups of several terms got 40 to 62% faster at p50: five popular terms took 1.7 ms instead of 4.4 ms on pubnet chunk 6465, and 1.9 ms instead of 4.6 ms on a chunk folded to seven times its density. Single-term lookups, and lookups on a volume already at its throughput limit, are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCzMKKrK3NstGKvaxsHhL2
tamirms
force-pushed
the
tamirms/events-split-terms
branch
from
October 5, 2026 16:19
9c001bd to
b7c7396
Compare
tamirms
force-pushed
the
tamirms/events-cold-read-concurrency
branch
from
October 5, 2026 16:19
1ba5ac5 to
462b735
Compare
tamirms
force-pushed
the
tamirms/events-split-terms
branch
from
October 8, 2026 08:18
b7c7396 to
9dd7c16
Compare
Contributor
Author
|
Squashed into #1086. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cold events readers were opened with a concurrency of 0, so each lookup
and fetch issued its reads one after another: one per term for a
lookup, and one per batch of a page's scattered events for a fetch. The
resolver now opens them with a concurrency of 8, which packfile's
ReadItems uses for index.pack and events.pack alike.
Measured on gp3 with a cold page cache, lookups of several terms got 40
to 62% faster at p50: five popular terms took 1.7 ms instead of 4.4 ms
on pubnet chunk 6465, and 1.9 ms instead of 4.6 ms on a chunk folded to
seven times its density. Single-term lookups, and lookups on a volume
already at its throughput limit, are unchanged.
🤖 Generated with Claude Code