Repository navigation
packfile: merge adjacent records into reads of up to 256 KiB - #1076
Conversation
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The implementation is coherent and well tested; only minor documentation wording needs correction.
Review effort: Balanced
Findings: 3
Open (3)
What changed in this PR
Updates packfile reads to merge adjacent records into size-based runs capped at 256 KiB.
Changes:
- Makes
ReadItemsconcurrency independent of read partitioning. - Reuses record scratch buffers and removes the 1 MiB buffer pool.
- Adds read-shape, cancellation, and error tests.
| File | Description |
|---|---|
design-docs/packfile-library.md |
Documents merged-read behavior. |
packfile/tail_test.go |
Makes read logging concurrency-safe. |
packfile/record.go |
Updates scratch-buffer lifecycle comments. |
packfile/record_test.go |
Revises alias-safety documentation. |
packfile/reader.go |
Implements 256 KiB merged reads. |
packfile/reader_test.go |
Removes superseded batching tests. |
packfile/reader_runs_test.go |
Tests read partitioning and failures. |
packfile/reader_bench_test.go |
Updates benchmark commentary. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // TestPassthroughDecodePreservesPayload pins that passthrough decode leaves | ||
| // rec.payload alone: a payload sharing scratch's array would make a later | ||
| // encoder decode write its output over its own input. |
7b92451 to
eb065d1
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Oversized reads can leave multi-gigabyte scratch buffers retained in the process-wide workspace pool.
4 open findings
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
| if cap(rec.scratch) < size { | ||
| rec.scratch = make([]byte, size) | ||
| } else { | ||
| rec.scratch = rec.scratch[:size] |
eb065d1 to
7ff5827
Compare
7ff5827 to
9519f06
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Oversized records can leave multi-gigabyte scratch buffers retained in the process-wide workspace pool.
4 open findings
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
There was a problem hiding this comment.
🟢 Approval recommended
The implementation is consistent and well-tested; only non-blocking read-ahead wording needs clarification.
4 open findings
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
9519f06 to
32873e6
Compare
| func (r *Reader) readRecords(rec *record, first, last int) ([]byte, error) { | ||
| start, end := r.offsets[first], r.offsets[last+1] | ||
| size := int(end - start) | ||
| if cap(rec.scratch) < size { | ||
| rec.scratch = make([]byte, size) | ||
| } else { | ||
| rec.scratch = rec.scratch[:size] | ||
| } | ||
| if _, err := r.file.ReadAt(rec.scratch, start); err != nil { | ||
| return nil, fmt.Errorf("packfile: read records [%d, %d]: %w", first, last, err) | ||
| } | ||
| return rec.scratch, nil | ||
| } |
32873e6 to
7853499
Compare
cjonas9
left a comment
There was a problem hiding this comment.
Overall, LGTM once #1075 lands + this gets rebased after! Two minor comments, one of them conditional on a comment I left in #1075, neither blocking.
I had the following results while reviewing this PR. They may be worth looking into more or benchmarking on EBS (I only checked locally), or just considering briefly:
- Warm-cache cost. Before, splitting positions across workers also split the decompression. Now one worker decompresses everything in its read, so when the data is already in memory, a lookup that needs fewer reads than there are workers gets slower with Concurrency 8. In a local benchmark (zstd records of ~10 KiB, data in the page cache), 12 adjacent records went from 41 to 65 µs and 40 adjacent records from 82 to 143 µs. That's small next to the savings on cold EBS reads, so the change is still right, but worth stating.
- Production concurrency.
resolve.goopens cold event readers with Concurrency 1 until #1086 raises it to 8, so the 23 → 13 ms result (measured with 8 reads in flight) applies once #1086 lands.
| continue | ||
| for first := firstRecord; first <= lastRecord; { | ||
| last := first | ||
| for last < lastRecord && r.offsets[last+2]-r.offsets[first] <= maxRunBytes { |
There was a problem hiding this comment.
Nit: the 256 KiB check counts record bytes, but the kernel reads whole 4 KiB pages. A read that starts partway into a page can touch 65 pages (260 KiB), and gp3 counts anything over 256 KiB as two I/Os. Rounding the read out to page boundaries before the check keeps every merged read at one I/O.
To that end, the change to consider is as follows:
- add helper:
const pageSize = 4 << 10
// pageSpan is the bytes the page cache reads to serve [start, end).
func pageSpan(start, end int64) int64 {
return (end+pageSize-1)&^(pageSize-1) - start&^(pageSize-1)
}
- then, here, for L452:
| for last < lastRecord && r.offsets[last+2]-r.offsets[first] <= maxRunBytes { | |
| for last < lastRecord && pageSpan(r.offsets[first], r.offsets[last+2]) <= maxRunBytes { |
- similarly, for L524:
if rec != last+1 || pageSpan(r.offsets[first], r.offsets[rec+1]) > maxRunBytes {
There was a problem hiding this comment.
Done in 120b7b4: both merge checks count the pages a read touches, so every merged read is one EBS I/O.
| buf, err := r.readRecords(rec, first, last) | ||
| if err != nil { | ||
| yield(nil, err) |
There was a problem hiding this comment.
This comment, which is contained in the dropdown below, is heavily conditional on this suggestion landing. If it does, the following should be considered
Follow-up changes
If the FADV_RANDOM change lands on #1075, ReadRange should prefetch its next read. Today the kernel sees ReadRange reading forward and fetches the next read in the background while the current one is decoded. FADV_RANDOM turns that off, so each read in a getLedgers or getTransactions page would wait for EBS in full. A FADV_WILLNEED hint keeps the overlap, and it only fetches bytes ReadRange is about to read:
| buf, err := r.readRecords(rec, first, last) | |
| if err != nil { | |
| yield(nil, err) | |
| buf, err := r.readRecords(rec, first, last) | |
| if err != nil { | |
| yield(nil, err) | |
| return | |
| } | |
| if last < lastRecord { // fetch the next read while this one is decoded | |
| next := r.offsets[last+1] | |
| end := min(max(r.offsets[last+2], next+maxRunBytes), r.offsets[lastRecord+1]) | |
| prefetch(r.file, next, end-next) | |
| } |
In fadvise_linux.go, next to adviseRandom:
// prefetch asks the kernel to start reading [offset, offset+n) without waiting.
func prefetch(f readAtCloser, offset, n int64) {
file, ok := f.(*os.File)
if !ok {
return
}
if sc, err := file.SyscallConn(); err == nil {
_ = sc.Control(func(fd uintptr) {
_ = unix.Fadvise(int(fd), offset, n, unix.FADV_WILLNEED) //nolint:gosec // G115: fd fits in int
})
}
}In fadvise_other.go:
func prefetch(readAtCloser, int64, int64) {}As before, this change is related, but adds an additional angle on this and #1075. So if it's better left to follow-up considerations after all of this lands, that's totally reasonable as well.
There was a problem hiding this comment.
Took this in 120b7b4, with one change: the prefetch starts from the second read rather than the first. getTransactions and unfiltered queryEvents ask ReadRange for a wide range and usually stop inside the first read, so prefetching from the first read would fetch up to 256 KiB they never use. A consumer that comes back for a second read is reading on, which is also when the kernel's readahead used to start. A read that spans more than one I/O also hints all of its 256 KiB pieces up front, since with readahead off the kernel would fetch them one at a time.
|
Both notes are in the description now: the warm-cache cost of merging, and that the 23 to 13 ms result needs #1086's 8 reads in flight. |
02961f0 to
120b7b4
Compare
The reader formed its reads by two rules at once. Adjacent records merged into one ReadAt of up to 1 MiB, and ReadItems also capped every read at ceil(positions / Concurrency) positions. With Concurrency above 1 the cap split runs of adjacent records: in a replay of pubnet queries with 8 reads in flight, one term's 156 adjacent index records went out as 6 to 8 small reads, and whole-chunk windows took 23 ms at p50 instead of 13 ms. The 1 MiB bound let a read grow past one EBS I/O: a cold 1 MiB read takes 10.9 ms on gp3 against 1.3 ms for 256 KiB. Adjacent records now merge by size alone into reads of up to 256 KiB, and a larger record is read alone. Concurrency changes how many of ReadItems' reads are in flight, not which reads it makes. ReadRange applies the same rule, one read at a time. Every read goes into the record workspace's scratch buffer, so the pool of 1 MiB read buffers and the one-off allocations for larger records are gone. Two older ReadItems tests are deleted; the new read-log tests cover their cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
The kernel's readahead fetched bytes no read asked for: past the end of reads it took for sequential, and past reads larger than the device's max I/O. Open now advises the file POSIX_FADV_RANDOM, so every read fetches exactly the pages it touches. Readahead also overlapped I/O with decoding, so that now comes from explicit POSIX_FADV_WILLNEED hints. ReadRange hints its next read from its second read on, never past the requested range, so a consumer that stops inside its first read costs no extra I/O. A read spanning more than one I/O, such as a large ledger, hints all of its 256 KiB pieces before it waits; without readahead the kernel would fetch them one at a time. Merged reads counted record bytes against the 256 KiB cap, but the page cache reads whole 4 KiB pages, so a read starting mid-page could touch 65 pages: two EBS I/Os. The cap now counts pages, and Open's first read starts on a page boundary, so the default 256 KiB tail read is one I/O too. Measured at the top of this stack through the daemon's handlers, on pubnet chunk 6504 on EBS gp3 with the page cache evicted before every request (900 requests per type): handler p50 for getLedgers of 10 ledgers went from 34.8 to 32.4 ms with 11% fewer bytes read, queryEvents on one contract from 4.39 to 3.83 ms, and queryEvents with no filter from 4.20 to 3.45 ms with 14% fewer bytes. getTransactions and getTransaction by hash were unchanged at p50. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
120b7b4 to
438e1d2
Compare
| readBufSize = 1 << 20 // 1 MiB pooled coalesced-read buffer | ||
| maxRunBytes = 256 << 10 // largest merged read: one EBS I/O | ||
| defaultFirstRead = 256 << 10 | ||
| pageSize = 4 << 10 // the page cache reads files in whole pages |



The problem
The reader sized its reads by two rules at once: adjacent records merged into one read of up to 1 MiB, and
ReadItemsalso split every read so that no worker took more than its share of positions. With 8 reads in flight the second rule broke up runs of adjacent records: in a replay of pubnet queries, one term's 156 adjacent index records went out as 6 to 8 small reads. And the first rule let a read grow past what one EBS I/O serves: a cold 1 MiB read takes 10.9 ms on gp3 against 1.3 ms for 256 KiB.On top of the reader's own reads, the kernel's readahead fetched bytes no read asked for: past the end of reads it took for sequential, and past reads larger than the device's max I/O.
The idea
Size every read for one EBS I/O and nothing else. Adjacent records merge by size alone into reads whose pages fit in 256 KiB, a larger record is read alone, and
Concurrencyonly decides how many of those reads are in flight, never which reads are made.ReadRangefollows the same rule one read at a time. Every read lands in the record workspace's scratch buffer, so the pool of 1 MiB buffers and the one-off allocations for large records are gone.Open advises each file
POSIX_FADV_RANDOM, so the kernel fetches exactly the pages a read touches, and Open's first read starts on a page boundary, so the default 256 KiB tail read stays one I/O. Where readahead used to overlap I/O with decoding, the reader now asks for it withPOSIX_FADV_WILLNEED:ReadRangehints its next read from its second read on, never past the requested range, and a read that spans more than one I/O hints all of its 256 KiB pieces before it waits.Measured
On the bench box's gp3 volume with a cold page cache, a 1 MiB read takes 10.9 ms and a 256 KiB read takes 1.3 ms.
In a replay of pubnet queries with 8 reads in flight, a lookup that reads 156 adjacent records went from 23 ms to 13 ms at p50: the old rules split them into 6 to 8 small reads, the new rule reads them in a few 256 KiB reads. The 8 reads in flight come from #1086; until then, cold event readers make one read at a time.
The readahead change, measured at the top of this stack through the daemon's handlers on pubnet chunk 6504, with the page cache evicted before every request and 900 requests per type. Times are spent inside the handler.
With the page cache warm, merging costs a little: one worker now decompresses a whole merged read instead of the read being split across workers. With 8 workers, a lookup of 12 adjacent records of about 10 KiB went from 41 to 65 µs, and 40 records from 82 to 143 µs. That is small next to the cold savings.
Two older
ReadItemstests are replaced by read-log tests that pin the exact reads.🤖 Generated with Claude Code