Skip to content

events: build the cold index from spilled slabs - #1098

Open
tamirms wants to merge 22 commits into
tamirms/events-hot-index-slabsfrom
tamirms/events-cold-index-runs
Open

tamirms wants to merge 22 commits into
tamirms/events-hot-index-slabsfrom
tamirms/events-cold-index-runs

Conversation

@tamirms

@tamirms tamirms commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

The problem

The cold events index is built at every freeze and in every backfill worker, and the build kept every term's bitmap for the whole chunk in memory until the chunk was done: about half a gigabyte on a pubnet chunk, 24 GB on an adversarial one, multiplied by the number of workers. The minimal perfect hash builder then wrote its own partition files under the temp directory.

The idea

Make the build an external sort. Postings go into the same fixed slab buffer the hot index uses. When a slab is done, its terms are written to disk as one sorted run, in the background while the next slab fills. At the end the runs are merged once, in a single streaming pass, into a terms file that both index files are written from. Because the runs are sorted by the term's routed key, the merged order is exactly the order the index files need.

Memory is bounded however many terms the chunk holds. While postings come in, it is at most two slabs. While the runs are merged, it is the merge's read buffers and the term being merged, and a term in every slab is held whole: about 66 MB for a term on every third sac-9000 event and about 1 GB at the event-id ceiling. While index.pack is written, it is at most one hash block of entries, of which it keeps only the short ones' bytes.

How it works

ColdIndexBuilder spills a run per full slab into a directory beside the index files (#1099 moves it to the local disk). The full slab goes to a goroutine that writes it while the next slab fills. At most one write is in flight, which keeps the runs in order, and a failed write fails the build at the next spill or at Write, which waits for any write in flight before it removes the runs.

The runs all go into one file in that directory, each after a 1 MiB hole, and the merge reads each run from its range of the file. So a build holds one open file however many runs it spills. A file per run would need one open file per 65,536 events: about 4,100 for a sac-9000 chunk, and up to 65,536 for a chunk dense in events, more than a host with a low hard limit on open files allows. The holes are there because readahead doesn't know where a run ends, so near the end of a run it reads ahead into whatever follows. A hole reads as zeros without a disk read, where the start of the next run, which the merge read long before, would be read from disk again once memory pressure had evicted it.

Write merges the runs once, into a terms file beside them that holds each term's index.pack entries, then builds index.hash with the sorted streamhash builder and index.pack from two reads of that file. A run's bitmaps are already in their final form, so a term that one slab holds is copied as it is. A term in several slabs is united, into reused bitmaps, and stays whole if the union takes at most 64 KiB. Otherwise it is split by slab, and each slab's bitmap is copied as it is. The merge reads the runs through buffers that share 16 MiB, between 4 KiB and 64 KiB per run, so they stay within 16 MiB up to 4,096 runs (268 million events) and take 4 KiB per run beyond that.

Roaring's Or steps through the receiver's keys one at a time, so uniting a term's slabs in slab order would cost the square of the slabs: each Or walks every key already in the union. The roaring replace moves to a fork commit where Or gallops past them instead, as And, AndNot and Xor already do. The fix is proposed upstream as RoaringBitmap/roaring#599. On a laptop, encoding a term in every slab, with 5 events per slab, takes 0.43 ms at a sac-9000 chunk's 4,120 slabs and 3.1 ms at 29,600, against 6.7 ms and 315 ms without it. The cold reader and the hot index unite a term's slabs the same way, so their lookups get the fix too.

index.pack stores each term's entries at the term's slot in the minimal perfect hash. Keys in order map to the hash's blocks in order, so streamhash bounds how far ahead of its slot an entry can arrive by MaxBlockKeys. The pack writer holds early entries in a ring of that size, each at its own slot, and refuses a slot outside the ring's reach as corrupt. The ring keeps an entry's bytes if it is at most 64 bytes long, and otherwise its place in the terms file, from which the writer reads it back when its slot comes, a split entry one slab at a time. Short entries are 97% of the pubnet chunk's below and all but 9,007 of the sac-9000 chunk's 90 million, and the kept bytes never took more than 72 KiB on either.

The runs are deleted after the merge, and the terms file once the index is written. They coexist only during the merge, where the build's scratch peaks at about three times the chunk's index.pack: 238 MB for the pubnet chunk below, whose index.pack is 80 MB, and 9.3 GB for the sac-9000 one, whose index.pack is 3.1 GB. If the build crashes they are left beside the chunk's "freezing" key, and the catalog sweep that destroys the key removes them as that key's scratch directory, so no new cleanup code was needed. The backfill walk derives each event's terms into one reused buffer, and the builder copies what it keeps. The whole-chunk map Bitmaps is now only a test fixture.

The backfill has no hot database, so the builder takes its postings from the raw ledgers. The freeze of a complete hot chunk, later in this stack, also hands it the hot index's sealed slabs as runs.

Measured

A real pubnet chunk with its diagnostic events (10,000 ledgers, 10.19 million events, 1.62 million terms), built from the bulk backend by bench ingest cold with one worker, twice each way, in the order before, after, after, before:

before after
whole build 65.6 and 66.2 s 61.2 and 60.9 s
events stage 32.1 and 32.8 s 27.8 and 27.6 s
peak RSS 1,496 and 1,557 MB 675 and 743 MB

Writing the index alone, from the same runs each time, for that chunk and for a sac-9000 chunk without its diagnostic events (270 million events in 4,120 slabs, 90 million terms), on local NVMe:

pubnet sac-9000
merge 1.1 s 60 s
index.hash 0.2 s 11 s
index.pack 1.9 s 102 s
whole Write 3.3 s 173 s
heap peak 33 MiB 63 MiB
scratch peak 238 MB 9.3 GB

The heap peak is the Go runtime's heap objects, sampled every millisecond, at their highest over several writes. The scratch peak is the runs and the terms file on disk at the end of the merge.

Merging 4,120 runs of synthetic terms (11 GB) on local NVMe, two merges each way:

merge a file per run one file
runs in the page cache 121 and 124 s 118 and 118 s
runs evicted 126 and 126 s 123 and 123 s
runs evicted, with 7 GB of page cache 128 and 131 s 128 and 129 s

Each evicted merge read the runs' 11.05 GB once. Without the holes, the one-file merge with 7 GB of page cache read 12.03 GB and took 139 s.

🤖 Generated with Claude Code

@tamirms
tamirms added this pull request to stack #1089 October 6, 2026 09:22
@tamirms
tamirms removed this pull request from stack #1089 October 8, 2026 08:18
@tamirms
tamirms force-pushed the tamirms/events-cold-index-runs branch from 39f9553 to 65cedca Compare October 8, 2026 08:18
@tamirms
tamirms added this pull request to stack #1102 October 8, 2026 08:19
@tamirms
tamirms force-pushed the tamirms/events-cold-index-runs branch from 65cedca to ecd07ab Compare October 8, 2026 08:34
@tamirms
tamirms force-pushed the tamirms/events-cold-index-runs branch 2 times, most recently from 038ba1a to 17e0837 Compare October 9, 2026 06:02
@tamirms
tamirms force-pushed the tamirms/events-cold-index-runs branch from 17e0837 to c004318 Compare October 15, 2026 19:15
@tamirms
tamirms force-pushed the tamirms/events-cold-index-runs branch from c004318 to 5d50b3a Compare October 19, 2026 13:00
@tamirms
tamirms force-pushed the tamirms/events-cold-index-runs branch 2 times, most recently from 9306de1 to bbc0019 Compare October 21, 2026 09:12
@tamirms
tamirms force-pushed the tamirms/events-cold-index-runs branch 2 times, most recently from c7e2d35 to 9b75e8e Compare October 21, 2026 09:52
@tamirms
tamirms force-pushed the tamirms/events-cold-index-runs branch 3 times, most recently from e582fed to e377ef3 Compare October 21, 2026 14:07
@tamirms
tamirms force-pushed the tamirms/events-cold-index-runs branch from e377ef3 to 2cd289c Compare October 21, 2026 14:43
@tamirms
tamirms force-pushed the tamirms/events-cold-index-runs branch 2 times, most recently from 6c9fba2 to 0df9481 Compare October 21, 2026 17:15
tamirms and others added 8 commits November 14, 2026 08:19
The cold events index was built from a map of every term of the chunk to
its bitmap, held in memory for the whole backfill or freeze of the chunk.
Its memory grew with the number of distinct terms in the chunk, once per
chunk being built.

ColdIndexBuilder keeps one slab of postings in memory, in a slab buffer of
the kind the hot index fills, keyed by the routed term key so that a
slab's terms come out in the order the MPHF builder wants. The slab is
written as a run file, in a directory beside the index files, when the
next slab starts or when the buffer fills, and Write merges the runs
three times: to count the terms, to build index.hash with the sorted
streamhash builder, and to write index.pack. Terms arrive in MPHF block
order and a block's slots are contiguous, so index.pack is written in
slot order holding back at most one block of entries. The runs are
removed once the index is written, or when the build is abandoned. A
build that crashes leaves them beside the chunk's "freezing" key, so the
catalog sweep removes the directory, as a scratch path of the kind, when
it destroys the key; a rebuild reads only the runs it wrote itself.

The MPHF is built with the sorted builder, so streamhash no longer
writes partition files under the temp directory. Bitmaps, the whole-chunk
map, is now a test fixture. The design documents describe the build and
the scratch directory.

Measured on the same real pubnet chunk (10,000 ledgers, 10.19M events)
built twice each way from the bulk backend: the chunk took 75.7 and 74.9 s
before, 77.1 and 77.2 s after; peak RSS 1,486 and 1,523 MB before, 730
and 727 MB after. On a chunk of 60M synthetic events the live heap at
28M events was 875 MB, of which the index held 10.5 MB, and the peak RSS
of 5.1 GB was reached before the index was written.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BiCL19kUrno3Z4SAYgpAfh
The cold build's final merge opened every spilled run with a 64 KiB
read buffer, and there is one run per 65,536 events, so the merge
held about a byte per event of the chunk. The runs now share 16 MiB of
read buffers, each between 4 KiB and 64 KiB. Pubnet chunks keep 64 KiB
per run; a chunk at the protocol's event ceiling drops from about
1.5 GB to about 100 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wwsd8qTEGeSb7poFEYcgg9
Write merged the spilled runs three times: to count the terms, to build
index.hash, and to write index.pack, decoding every slab's bitmap into a
new bitmap on the last pass. It now merges them once, into a terms file
beside the runs that holds each term's index.pack entries, and builds
index.hash and index.pack from two reads of that file. A run's bitmaps
are already in their final form, so a term in one slab, and each slab of
a split term, is copied as it is; only a term in several slabs that stays
whole is united, into reused bitmaps. The artifacts do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjQed1VX9uGU9PWpAjBiz7
The cold index builder wrote each full slab as a run on the walk's
goroutine, so every ledger that filled a slab waited for the write. It now
hands the full slab to a goroutine and fills a second one meanwhile. At
most one write is in flight, which keeps the runs in order, and a failed
write fails the build at its next spill or at finalize.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjQed1VX9uGU9PWpAjBiz7
The pack pass held every entry that arrived before its slot in a heap
ordered by slot, so most entries paid a push and a pop. Keys in order map
to MPHF blocks in order, so streamhash bounds how far ahead a slot can
arrive by MaxBlockKeys. The writer now holds early entries in a ring of
that size, each at its own slot, and refuses a slot out of the ring's
reach as corrupt. The pack's bytes do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjQed1VX9uGU9PWpAjBiz7
The backfill walk derived each event's terms into a slice of its own. It
now derives them into one buffer it reuses; the index builder copies what
it keeps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjQed1VX9uGU9PWpAjBiz7
The cold index build spilled each slab's terms to a run file of its own
and opened every run at once to merge them, so a build needed one open
file per 65,536 events. A chunk dense in events could need more open
files than the process may hold, and its build then failed the same way
on every attempt.

Each spill now appends its run to one file and records where the run
starts and how long it is. The merge reads each run from its range of
that file. A build holds one file open however many runs it spills, and
writes and reads the same bytes as before.

Each run after the first starts after a 1 MiB hole. Readahead doesn't
know where a run ends, so it reads past each run's end. Into a hole that
costs no disk reads; into the next run's start, which the merge read
long before, it reads those pages from disk again once memory pressure
has evicted them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AqbUZNjCE6UXuYDeTwEiWb
Write removes the runs on every return, but a Write canceled before its
own spill returned without waiting for a spill an earlier Add had
started. That spill could still be writing the runs as they were
removed, and could create their directory again after it was gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AqbUZNjCE6UXuYDeTwEiWb
@tamirms
tamirms force-pushed the tamirms/events-cold-index-runs branch from 52395d5 to 02d5138 Compare November 14, 2026 08:41
tamirms and others added 2 commits November 16, 2026 07:35
Merging the runs into the terms file and writing index.pack from it read
and write the whole index on disk, about two minutes for a sac-9000
chunk, and neither checked the build's context, so a cancelled freeze or
backfill ran them to the end first. Both now check it every 4,096 terms.

Add also refuses an event in an earlier slab than the one it is filling,
whose run would otherwise land out of slab order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes
…uffer

An event has at most maxTermsPerEvent terms and the buffer holds that many
postings for each of a slab's events, so a slab is always one run. Add
checked for a full buffer on every key and the term encoder united a slab
spilled in several runs; both go, and Add refuses an event with more terms
than an event can carry instead.

Also: spill no longer takes the next slab's number, which Write passed as
a placeholder 0, so Add resets the slab where the slab changes; the
builder counts its slabs instead of tracking the largest id; the index.pack
writer holds an early entry or writes the next and drains in two steps;
Close, isEmpty and numKeys go back where they were in cold_format.go; and
four dense comments read plainly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Pending payload retention can consume GiB per worker, bitmap aggregation scales quadratically with slab count, and missing per-key cancellation checks can substantially delay shutdown.

3 open issues
What changed in this PR

Replaces whole-chunk event bitmap accumulation with a disk-backed cold-index build, reducing memory pressure during backfill and freeze while preserving the cold index format.

Changes:

  • Spill reusable posting slabs into sorted runs, merge them, and build the hash and pack files from the merged terms.
  • Integrate the builder into cold ingestion and remove abandoned build scratch through catalog sweeping.
  • Extend tests and document scratch placement, memory usage, and disk requirements.
File Description
design-docs/​getevents-full-history-design.md Documents slab spilling and index finalization.
design-docs/​full-history-streaming-workflow.md Documents scratch lifecycle and disk requirements.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​match_test.go Uses the replacement index fixture helper.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​hot_index.go Adds slab reset for buffer reuse.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_writer.go Updates the builder reference in comments.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_split_test.go Adapts split-index tests to the new interfaces.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_runs.go Implements shared-file spills and streaming merge.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_runs_test.go Tests merging, truncation, and shared-file storage.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_reader.go Adopts the renamed batch lookup method.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_reader_test.go Updates index fixture construction and comments.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_index.go Implements the slab-backed builder and slot-order writer.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_index_test.go Adds builder, cleanup, encoding, and slot-validation tests.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_format.go Uses sorted hash construction and updates lookup interfaces.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_format_test.go Adapts hash tests to sorted input and batch lookup.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​bitmaps.go Removes the production whole-chunk bitmap map.
cmd/​stellar-rpc/​internal/​rpcv2/​ingest/​metrics.go Updates a metric-stage comment.
cmd/​stellar-rpc/​internal/​rpcv2/​ingest/​ingester.go Updates failure-contract terminology in comments.
cmd/​stellar-rpc/​internal/​rpcv2/​ingest/​ingest_test.go Updates failure assertions and builder terminology.
cmd/​stellar-rpc/​internal/​rpcv2/​ingest/​events.go Feeds the new builder and reuses term buffers.
cmd/​stellar-rpc/​internal/​rpcv2/​geometry/​paths.go Defines event-index scratch paths.
cmd/​stellar-rpc/​internal/​rpcv2/​geometry/​keys_test.go Verifies scratch-path mapping.
cmd/​stellar-rpc/​internal/​rpcv2/​catalog/​catalog_sweep.go Removes build scratch when destroying artifacts.
cmd/​stellar-rpc/​internal/​rpcv2/​catalog/​catalog_sweep_test.go Tests cleanup of crash-left scratch.

Review effort: Balanced

tamirms and others added 7 commits November 16, 2026 09:00
Close, isEmpty and numKeys moved back to the end of cold_format.go put the
unexported lookupRouted ahead of the exported Close, which funcorder
rejects. This restores the order this PR had.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes
The index.pack writer kept the bytes of every entry that arrived before its
slot until the slot came. A hash block holds about 31,600 terms whose slots
come in no useful order, so the writer held nearly a whole block, and an
index of fewer than about 63,000 terms has only two blocks: for a sac-9000
chunk's roughly 42,000 terms, about half the index.

The ring now keeps where a held entry is in the terms file and reads it
back when its slot comes, so the writer holds no entry bytes. Reading an
entry back costs about 0.8 us from the page cache, about 1.2 s over a
pubnet chunk's 1.6 million terms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes
…n on every term

A review of the build found that it still moved a big term as one piece in
several places and checked for cancellation too rarely:

- The merge copied every run's bitmap of a term into one buffer. It now
  yields the run readers' own buffers, which is also about 10% faster.
- A split term was assembled whole before going to the terms file and read
  back whole before going to index.pack. Both now go one slab at a time,
  and the hash and pack passes read only the terms file's keys, fetching
  each entry by its position.
- The build checked its context every 4,096 terms, which at sac-9000 is
  millions of entries apart. It now checks on every term, in the merge, the
  hash pass and the pack write.
- An index needing more than 2^32 index.pack items failed only when
  index.pack was finished. The build now fails as soon as the merge crosses
  that count.
- Spills run on the hot index's sealer instead of their own copy of it.

On a chunk of three terms in every one of 29,600 slabs, peak heap went from
1,316 to 416 MB in the merge, 526 to 7 MB in the hash pass, and 245 to 12 MB
in the pack write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes
The runs and the terms file they merge into take about 4 to 6 times the
chunk's index.pack, not two, and the merge keeps about 12 KB of read buffers
per run on top of the two slabs of postings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes
…h each

A cleanup review of the build found repeated mechanics and a bound looser
than the format:

- The head of a runs-file or terms-file entry, key || uvarint(len), was
  written by hand in three places, each moving the key and the length to
  the heap. writeHead appends them into the writer's own buffer, with no
  allocation per entry.
- A split term's size comes from its parts rather than a second walk over
  every slab, and the slab count comes from the runs rather than a copy kept
  by Add, which removes a consistency check that could not fail.
- Write's cleanup calls Close, and coldRuns.remove calls removeRuns, so each
  is written once.
- The merge heap orders runs by slab directly, without a wrapper type or a
  run index.
- Run values and split entries are bounded by slabEntryMaxBytes, the largest
  slab entry the format allows, instead of a 1 MiB constant.
- The union of a term's slabs is no longer run-optimized: each slab's bitmap
  already is, and slabs never share a container.
- Both slabs are released once the last one is spilled, before the merge.
- Reads return early instead of chaining error checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes
roaring's Or stepped through the receiver one key at a time, so building a
bitmap one slab at a time in key order was quadratic in the number of
slabs. The index build's unions and both readers' assembly of a term across
slabs do exactly that: for a term in every slab, 6.7 ms at a sac-9000
chunk's 4,120 slabs and 315 ms at 29,600. Our fork's commit now gallops past
the receiver's keys the way And, AndNot and Xor already do (proposed
upstream as RoaringBitmap/roaring#599): 0.43 ms and 3.1 ms. The pinned
commit also moves #566 and #572 onto roaring's current master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes
writeHead converts the length it is given, which gosec cannot see is never
negative, where the code it replaced converted len() directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The persisted cold-index rewrite introduces asynchronous ownership and spill cleanup whose query correctness, crash recovery, and worst-case memory bounds require runtime validation beyond this source-only review.

2 open issues
1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The cold index issues a separate file read for each small term and continues merging after buffered-write failures, causing excessive I/O and delaying scratch-file cleanup.

4 open issues

Review effort: Max

tamirms and others added 4 commits November 16, 2026 11:54
The terms file's writes ignored errors until the final flush, so after a
disk-full error the merge still read and decoded every remaining run before
the build failed and removed its scratch. The write helpers now return the
error a bufio.Writer keeps, so the merge stops at the next term.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes
packfile already refuses more items than index.pack can count when the
write finishes, so the merge's own count only failed such a chunk sooner,
and it had to know how the pack lays a split term out. No real chunk comes
near the limit. With the count gone, the encoder no longer reports whether
it split a term.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes
gosec sees the bound checked before each of these conversions, so
nolintlint reports the directives as unused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes
The index.pack writer read every entry back from the terms file when its
slot came, one read per term. Most terms are short: 97% of a real pubnet
chunk's 1.62 million entries are 64 bytes or less. The writer now keeps
those, which its scan of the terms file reads anyway, and reads back only
longer entries. On that chunk the reads fell from 1,617,471 to 53,962 and
the index.pack write from 2.78 s to 1.93 s, with identical files. The kept
bytes are at most one hash block's short entries: 72 KiB at peak there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human review is needed to assess an unpublished moderate-severity finding that discarded spill-write errors can delay failure reporting and scratch-file cleanup.

2 open issues
2 resolved since last review

Review effort: Max

The runs and the terms file peak at about three times the chunk's
index.pack: 238 MB against 80 MB on a pubnet chunk, and 9.3 GB against
3.1 GB on a sac-9000 chunk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9U6SAy9FhAyaWWJRQ5bes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants