Skip to content

packfile: merge adjacent records into reads of up to 256 KiB - #1076

Merged
tamirms merged 2 commits into
feature/full-historyfrom
tamirms/packfile-merged-reads
Oct 21, 2026
Merged

tamirms merged 2 commits into
feature/full-historyfrom
tamirms/packfile-merged-reads

Conversation

@tamirms

@tamirms tamirms commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

The problem

The reader sized its reads by two rules at once: adjacent records merged into one read of up to 1 MiB, and ReadItems also split every read so that no worker took more than its share of positions. With 8 reads in flight the second rule broke up runs of adjacent records: in a replay of pubnet queries, one term's 156 adjacent index records went out as 6 to 8 small reads. And the first rule let a read grow past what one EBS I/O serves: a cold 1 MiB read takes 10.9 ms on gp3 against 1.3 ms for 256 KiB.

On top of the reader's own reads, the kernel's readahead fetched bytes no read asked for: past the end of reads it took for sequential, and past reads larger than the device's max I/O.

The idea

Size every read for one EBS I/O and nothing else. Adjacent records merge by size alone into reads whose pages fit in 256 KiB, a larger record is read alone, and Concurrency only decides how many of those reads are in flight, never which reads are made. ReadRange follows the same rule one read at a time. Every read lands in the record workspace's scratch buffer, so the pool of 1 MiB buffers and the one-off allocations for large records are gone.

Open advises each file POSIX_FADV_RANDOM, so the kernel fetches exactly the pages a read touches, and Open's first read starts on a page boundary, so the default 256 KiB tail read stays one I/O. Where readahead used to overlap I/O with decoding, the reader now asks for it with POSIX_FADV_WILLNEED: ReadRange hints its next read from its second read on, never past the requested range, and a read that spans more than one I/O hints all of its 256 KiB pieces before it waits.

Measured

On the bench box's gp3 volume with a cold page cache, a 1 MiB read takes 10.9 ms and a 256 KiB read takes 1.3 ms.

In a replay of pubnet queries with 8 reads in flight, a lookup that reads 156 adjacent records went from 23 ms to 13 ms at p50: the old rules split them into 6 to 8 small reads, the new rule reads them in a few 256 KiB reads. The 8 reads in flight come from #1086; until then, cold event readers make one read at a time.

The readahead change, measured at the top of this stack through the daemon's handlers on pubnet chunk 6504, with the page cache evicted before every request and 900 requests per type. Times are spent inside the handler.

request p50 p99 bytes read
getLedgers, 10 ledgers 34.8 → 32.4 ms 62.1 → 54.7 ms −11%
getLedgers, 20 ledgers 69.4 → 62.9 ms 122.4 → 110.7 ms −6%
getTransactions, 200 7.29 → 7.21 ms 12.25 → 12.27 ms −9%
queryEvents, one contract 4.39 → 3.83 ms 5.31 → 5.32 ms 0%
queryEvents, no filter, 100 4.20 → 3.45 ms 5.91 → 4.35 ms −14%
getTransaction by hash 6.24 → 6.22 ms 9.56 → 8.35 ms −5%

With the page cache warm, merging costs a little: one worker now decompresses a whole merged read instead of the read being split across workers. With 8 workers, a lookup of 12 adjacent records of about 10 KiB went from 41 to 65 µs, and 40 records from 82 to 143 µs. That is small next to the cold savings.

Two older ReadItems tests are replaced by read-log tests that pin the exact reads.

🤖 Generated with Claude Code

Copilot AI balanced review requested due to automatic review settings October 5, 2026 10:54
@tamirms
tamirms added this pull request to stack #1081 October 5, 2026 10:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation is coherent and well tested; only minor documentation wording needs correction.

Review effort: Balanced
Findings: 3 Low severity

Open (3)
What changed in this PR

Updates packfile reads to merge adjacent records into size-based runs capped at 256 KiB.

Changes:

  • Makes ReadItems concurrency independent of read partitioning.
  • Reuses record scratch buffers and removes the 1 MiB buffer pool.
  • Adds read-shape, cancellation, and error tests.
File Description
design-docs/​packfile-library.md Documents merged-read behavior.
packfile/​tail_test.go Makes read logging concurrency-safe.
packfile/​record.go Updates scratch-buffer lifecycle comments.
packfile/​record_test.go Revises alias-safety documentation.
packfile/​reader.go Implements 256 KiB merged reads.
packfile/​reader_test.go Removes superseded batching tests.
packfile/​reader_runs_test.go Tests read partitioning and failures.
packfile/​reader_bench_test.go Updates benchmark commentary.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +246 to +248
// TestPassthroughDecodePreservesPayload pins that passthrough decode leaves
// rec.payload alone: a payload sharing scratch's array would make a later
// encoder decode write its output over its own input.
Comment thread design-docs/packfile-library.md Resolved
@tamirms
tamirms removed this pull request from stack #1081 October 5, 2026 16:19
@tamirms
tamirms added this pull request to stack #1089 October 5, 2026 16:24
@tamirms tamirms added this to the platform sprint 76 milestone Oct 6, 2026
@tamirms
tamirms removed this pull request from stack #1089 October 8, 2026 08:18
@tamirms
tamirms force-pushed the tamirms/packfile-merged-reads branch from 7b92451 to eb065d1 Compare October 8, 2026 08:18
Copilot AI balanced review requested due to automatic review settings October 8, 2026 08:18
@tamirms
tamirms added this pull request to stack #1102 October 8, 2026 08:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Oversized reads can leave multi-gigabyte scratch buffers retained in the process-wide workspace pool.

4 open findings

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Comment on lines +643 to +646
if cap(rec.scratch) < size {
rec.scratch = make([]byte, size)
} else {
rec.scratch = rec.scratch[:size]
Copilot AI balanced review requested due to automatic review settings October 8, 2026 08:34
@tamirms
tamirms force-pushed the tamirms/packfile-merged-reads branch from eb065d1 to 7ff5827 Compare October 8, 2026 08:34
@tamirms
tamirms force-pushed the tamirms/packfile-merged-reads branch from 7ff5827 to 9519f06 Compare October 8, 2026 08:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Oversized records can leave multi-gigabyte scratch buffers retained in the process-wide workspace pool.

4 open findings

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Copilot AI balanced review requested due to automatic review settings October 8, 2026 08:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation is consistent and well-tested; only non-blocking read-ahead wording needs clarification.

4 open findings

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Copilot AI balanced review requested due to automatic review settings October 9, 2026 06:02
@tamirms
tamirms force-pushed the tamirms/packfile-merged-reads branch from 9519f06 to 32873e6 Compare October 9, 2026 06:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +640 to +652
func (r *Reader) readRecords(rec *record, first, last int) ([]byte, error) {
start, end := r.offsets[first], r.offsets[last+1]
size := int(end - start)
if cap(rec.scratch) < size {
rec.scratch = make([]byte, size)
} else {
rec.scratch = rec.scratch[:size]
}
if _, err := r.file.ReadAt(rec.scratch, start); err != nil {
return nil, fmt.Errorf("packfile: read records [%d, %d]: %w", first, last, err)
}
return rec.scratch, nil
}
@tamirms
tamirms requested a review from Shaptic October 12, 2026 20:35
Copilot AI balanced review requested due to automatic review settings October 15, 2026 18:13
@tamirms
tamirms force-pushed the tamirms/packfile-merged-reads branch from 32873e6 to 7853499 Compare October 15, 2026 18:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Static review identified no blocking issues in read grouping, buffer ownership, or worker dispatch; tests were not executed.

5 open findings

🧠 Review effort: Balanced

@cjonas9 cjonas9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, LGTM once #1075 lands + this gets rebased after! Two minor comments, one of them conditional on a comment I left in #1075, neither blocking.

I had the following results while reviewing this PR. They may be worth looking into more or benchmarking on EBS (I only checked locally), or just considering briefly:

  • Warm-cache cost. Before, splitting positions across workers also split the decompression. Now one worker decompresses everything in its read, so when the data is already in memory, a lookup that needs fewer reads than there are workers gets slower with Concurrency 8. In a local benchmark (zstd records of ~10 KiB, data in the page cache), 12 adjacent records went from 41 to 65 µs and 40 adjacent records from 82 to 143 µs. That's small next to the savings on cold EBS reads, so the change is still right, but worth stating.
  • Production concurrency. resolve.go opens cold event readers with Concurrency 1 until #1086 raises it to 8, so the 23 → 13 ms result (measured with 8 reads in flight) applies once #1086 lands.

continue
for first := firstRecord; first <= lastRecord; {
last := first
for last < lastRecord && r.offsets[last+2]-r.offsets[first] <= maxRunBytes {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the 256 KiB check counts record bytes, but the kernel reads whole 4 KiB pages. A read that starts partway into a page can touch 65 pages (260 KiB), and gp3 counts anything over 256 KiB as two I/Os. Rounding the read out to page boundaries before the check keeps every merged read at one I/O.

To that end, the change to consider is as follows:

  • add helper:
const pageSize = 4 << 10

// pageSpan is the bytes the page cache reads to serve [start, end).
func pageSpan(start, end int64) int64 {
	return (end+pageSize-1)&^(pageSize-1) - start&^(pageSize-1)
}
  • then, here, for L452:
Suggested change
for last < lastRecord && r.offsets[last+2]-r.offsets[first] <= maxRunBytes {
for last < lastRecord && pageSpan(r.offsets[first], r.offsets[last+2]) <= maxRunBytes {
  • similarly, for L524:
			if rec != last+1 || pageSpan(r.offsets[first], r.offsets[rec+1]) > maxRunBytes {

@tamirms tamirms Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 120b7b4: both merge checks count the pages a read touches, so every merged read is one EBS I/O.

Comment on lines +455 to +457
buf, err := r.readRecords(rec, first, last)
if err != nil {
yield(nil, err)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment, which is contained in the dropdown below, is heavily conditional on this suggestion landing. If it does, the following should be considered

Follow-up changes

If the FADV_RANDOM change lands on #1075, ReadRange should prefetch its next read. Today the kernel sees ReadRange reading forward and fetches the next read in the background while the current one is decoded. FADV_RANDOM turns that off, so each read in a getLedgers or getTransactions page would wait for EBS in full. A FADV_WILLNEED hint keeps the overlap, and it only fetches bytes ReadRange is about to read:

Suggested change
buf, err := r.readRecords(rec, first, last)
if err != nil {
yield(nil, err)
buf, err := r.readRecords(rec, first, last)
if err != nil {
yield(nil, err)
return
}
if last < lastRecord { // fetch the next read while this one is decoded
next := r.offsets[last+1]
end := min(max(r.offsets[last+2], next+maxRunBytes), r.offsets[lastRecord+1])
prefetch(r.file, next, end-next)
}

In fadvise_linux.go, next to adviseRandom:

// prefetch asks the kernel to start reading [offset, offset+n) without waiting.
func prefetch(f readAtCloser, offset, n int64) {
	file, ok := f.(*os.File)
	if !ok {
		return
	}
	if sc, err := file.SyscallConn(); err == nil {
		_ = sc.Control(func(fd uintptr) {
			_ = unix.Fadvise(int(fd), offset, n, unix.FADV_WILLNEED) //nolint:gosec // G115: fd fits in int
		})
	}
}

In fadvise_other.go:

func prefetch(readAtCloser, int64, int64) {}

As before, this change is related, but adds an additional angle on this and #1075. So if it's better left to follow-up considerations after all of this lands, that's totally reasonable as well.

@tamirms tamirms Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took this in 120b7b4, with one change: the prefetch starts from the second read rather than the first. getTransactions and unfiltered queryEvents ask ReadRange for a wide range and usually stop inside the first read, so prefetching from the first read would fetch up to 256 KiB they never use. A consumer that comes back for a second read is reading on, which is also when the kernel's readahead used to start. A read that spans more than one I/O also hints all of its 256 KiB pieces up front, since with readahead off the kernel would fetch them one at a time.

@tamirms

tamirms commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Both notes are in the description now: the warm-cache cost of merging, and that the 23 to 13 ms result needs #1086's 8 reads in flight.

@tamirms
tamirms force-pushed the tamirms/packfile-merged-reads branch from 02961f0 to 120b7b4 Compare October 21, 2026 09:12
Base automatically changed from tamirms/packfile-tail-reads to feature/full-history October 21, 2026 09:31
tamirms and others added 2 commits October 21, 2026 10:31
The reader formed its reads by two rules at once. Adjacent records
merged into one ReadAt of up to 1 MiB, and ReadItems also capped every
read at ceil(positions / Concurrency) positions. With Concurrency above
1 the cap split runs of adjacent records: in a replay of pubnet queries
with 8 reads in flight, one term's 156 adjacent index records went out
as 6 to 8 small reads, and whole-chunk windows took 23 ms at p50 instead
of 13 ms. The 1 MiB bound let a read grow past one EBS I/O: a cold 1 MiB
read takes 10.9 ms on gp3 against 1.3 ms for 256 KiB.

Adjacent records now merge by size alone into reads of up to 256 KiB,
and a larger record is read alone. Concurrency changes how many of
ReadItems' reads are in flight, not which reads it makes. ReadRange
applies the same rule, one read at a time. Every read goes into the
record workspace's scratch buffer, so the pool of 1 MiB read buffers and
the one-off allocations for larger records are gone.

Two older ReadItems tests are deleted; the new read-log tests cover
their cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
The kernel's readahead fetched bytes no read asked for: past the end of
reads it took for sequential, and past reads larger than the device's
max I/O. Open now advises the file POSIX_FADV_RANDOM, so every read
fetches exactly the pages it touches.

Readahead also overlapped I/O with decoding, so that now comes from
explicit POSIX_FADV_WILLNEED hints. ReadRange hints its next read from
its second read on, never past the requested range, so a consumer that
stops inside its first read costs no extra I/O. A read spanning more
than one I/O, such as a large ledger, hints all of its 256 KiB pieces
before it waits; without readahead the kernel would fetch them one at
a time.

Merged reads counted record bytes against the 256 KiB cap, but the page
cache reads whole 4 KiB pages, so a read starting mid-page could touch
65 pages: two EBS I/Os. The cap now counts pages, and Open's first read
starts on a page boundary, so the default 256 KiB tail read is one I/O
too.

Measured at the top of this stack through the daemon's handlers, on
pubnet chunk 6504 on EBS gp3 with the page cache evicted before every
request (900 requests per type): handler p50 for getLedgers of 10
ledgers went from 34.8 to 32.4 ms with 11% fewer bytes read, queryEvents
on one contract from 4.39 to 3.83 ms, and queryEvents with no filter
from 4.20 to 3.45 ms with 14% fewer bytes. getTransactions and
getTransaction by hash were unchanged at p50.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tamirms
tamirms force-pushed the tamirms/packfile-merged-reads branch from 120b7b4 to 438e1d2 Compare October 21, 2026 09:31
@tamirms
tamirms requested a balanced review from Copilot October 21, 2026 09:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

readBufSize = 1 << 20 // 1 MiB pooled coalesced-read buffer
maxRunBytes = 256 << 10 // largest merged read: one EBS I/O
defaultFirstRead = 256 << 10
pageSize = 4 << 10 // the page cache reads files in whole pages
@tamirms
tamirms merged commit 953afa5 into main Oct 21, 2026
17 checks passed
@tamirms
tamirms deleted the tamirms/packfile-merged-reads branch October 21, 2026 09:51
@github-project-automation github-project-automation Bot moved this from Needs Review to Done in Platform Scrum Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants