Repository navigation
packfile: reuse Open's first-read buffer - #1046
Merged
tamirms merged 0 commit intoOct 5, 2026
Merged
Conversation
tamirms
force-pushed
the
tamirms/packfile-reuse-buffers
branch
from
October 5, 2026 10:53
ec9e7da to
0872c4c
Compare
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
tamirms
changed the base branch from
feature/full-history
to
tamirms/packfile-byte-limit
October 5, 2026 10:55
tamirms
force-pushed
the
tamirms/packfile-reuse-buffers
branch
from
October 5, 2026 16:19
0872c4c to
1a38a80
Compare
tamirms
force-pushed
the
tamirms/packfile-byte-limit
branch
from
October 5, 2026 16:19
b7a055b to
6ab0f8d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every request opens its packs, and each open reads the end of the file
into a new buffer of FirstRead bytes: 256 KiB by default and for
events.pack, 64 KiB for ledger packs. The Reader keeps a copy of the
tail or the second read's buffer, never the first read's, so that
buffer is garbage once the open finishes. It is most of what an open
allocates: 262 of the 267 KB that BenchmarkOpen's Open, TotalItems and
Close allocated at 12,600 records, with the file in the page cache.
The first read now goes into the scratch buffer of a workspace from
recordWorkspacePool, which already holds every read's buffer, and the
workspace goes back to the pool when the open finishes, whether or not
it succeeded. A scratch buffer smaller than the first read is replaced
by one of the right size, so the pool's buffers settle at the largest
read the process makes. At 12,600, 80,000 and 450,000 records the
benchmark now takes 35, 43 and 74 µs instead of 55, 67 and 94 (medians
of 10 interleaved runs), and allocates 5.5, 28 and 143 KB instead of
267, 288 and 402.
The pool needs no cap: the buffer's size is FirstRead(fileSize), which
the store sets and the file size bounds, so no file's contents can make
the pool keep a large buffer. What an open still allocates is mostly the
tail the Reader keeps, which is not pooled because AppData returns a
view into it that a caller may hold past Close.
🤖 Generated with Claude Code