Repository navigation
packfile: read only the missing tail bytes at open - #1075
Conversation
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The implementation is localized, preserves default behavior, and thoroughly tests boundary and fallback cases.
Review effort: Balanced
Findings: None
What changed in this PR
Optimizes packfile opening by reading only missing tail bytes and tuning ledger cold-pack reads.
Changes:
- Adds configurable first-read sizing.
- Avoids rereading already-fetched tail data.
- Uses 64 KiB first reads for ledger cold packs and adds coverage.
| File | Description |
|---|---|
design-docs/packfile-library.md |
Documents first-read behavior. |
cmd/stellar-rpc/internal/rpcv2/stores/ledger/cold_reader.go |
Configures 64 KiB ledger tail reads. |
cmd/stellar-rpc/internal/rpcv2/packfile/tail_test.go |
Tests read boundaries and fallback behavior. |
cmd/stellar-rpc/internal/rpcv2/packfile/reader.go |
Implements configurable, non-overlapping tail reads. |
cmd/stellar-rpc/internal/rpcv2/packfile/packfile.go |
Updates validation documentation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
a6444df to
b1b0904
Compare
There was a problem hiding this comment.
🟢 Approval recommended
The implementation matches the design and covers boundary and fallback behavior with focused tests.
0 open findings
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
b1b0904 to
4cf054b
Compare
There was a problem hiding this comment.
🟢 Approval recommended
The implementation matches the described optimization and covers its boundary cases with focused tests.
0 open findings
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
4cf054b to
2db23ff
Compare
2db23ff to
05619c3
Compare
Open reads the last 256 KiB of every pack. When the index, app data and trailer are larger, it reads the whole index region again, up to 256 KiB more than it needs; events.pack's index reaches 0.8 to 1 MB on large chunks. For a ledger pack 256 KiB is too much: its tail is under 42 KB (16 to 28 KB on real chunks), and a request opens every chunk it routes to. ReaderOptions.FirstRead now sizes Open's first read from the file size, so a store whose tail grows with the file can follow it; nil keeps 256 KiB. When the first read falls short of the tail, Open reads only the missing bytes in front of it. The ledger cold reader sets 64 KiB, which always covers its tail. doOpen now hands the opened file to openFile, so a test can record every read Open makes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
05619c3 to
0a49615
Compare
cjonas9
left a comment
There was a problem hiding this comment.
Overall, this LGTM! I have a few comments and thoughts, but none are worth blocking over.
Heads up: I'm benchmarking this PR on my own RPCv2 node (EBS) to see if adding FADV_RANDOM on the packfile fd at open gives a substantial drop in bytes read per request.
Why: this PR's win is fewer bytes per open on EBS. The record reads after the open use the same fd, and kernel readahead can pull in bytes nobody asked for on top of the coalescing #1076 already does. If that overshoot is as big as the tail saving, it's a bigger lever in the same open path and worth knowing before we size this win.
Numbers across three seeds in a couple of hours.
|
|
||
| // coldFirstRead covers a cold pack's whole tail: a chunk's index is under | ||
| // 42 KB at any ledger size, and the app data and trailer under 100 bytes. | ||
| const coldFirstRead = 64 << 10 |
There was a problem hiding this comment.
Imo, I think there's a good case for deriving this instead of hard-coding it. Neither reason is crucial, but I think it's a fairly low-cost change:
- this number could drift over time. We have a lot of buffer now between ~40KB and 64KB, packfile: decode only the index groups a read touches #1077's 17-byte directory entries make it 41,823 B. If packfile exports those sizes, coldFirstRead becomes a constant expression (rounded up to a page) that moves with the format, and nobody has to re-check the 64 KiB claim when the index layout changes
- it trims 16KB per open. My back-of-the-envelope math makes the improvement seem fairly modest, but a bonus nonetheless
Thoughts?
There was a problem hiding this comment.
Thanks for working through the numbers. I'd keep 64 KiB. Deriving it from today's worst case (41,823 B, so 44 KiB rounded to a page) saves about 20 KiB inside the same single read. The headroom is there on purpose: the ledger layout planned for LCM V3 puts typical tails at about 45 to 53 KB, past what a derived value would allow. And a tail that outgrows the first read costs one extra read, not a failure. Deriving it would also mean exporting a worst-case size formula from packfile for this one caller.
| fi, err := f.Stat() | ||
| if err != nil { | ||
| _ = f.Close() | ||
| return openResult{err: fmt.Errorf("packfile: stat: %w", err)} | ||
| } | ||
| fileSize := fi.Size() | ||
| res := openFile(f, fi.Size(), firstRead) | ||
| if res.err != nil { | ||
| _ = f.Close() | ||
| } | ||
| return res |
There was a problem hiding this comment.
| fi, err := f.Stat() | |
| if err != nil { | |
| _ = f.Close() | |
| return openResult{err: fmt.Errorf("packfile: stat: %w", err)} | |
| } | |
| fileSize := fi.Size() | |
| res := openFile(f, fi.Size(), firstRead) | |
| if res.err != nil { | |
| _ = f.Close() | |
| } | |
| return res | |
| var res openResult | |
| // Closes f unless res took it; a panic in openFile leaves res empty, so it closes then too. | |
| defer func() { | |
| if res.file == nil { | |
| _ = f.Close() | |
| } | |
| }() | |
| fi, err := f.Stat() | |
| if err != nil { | |
| return openResult{err: fmt.Errorf("packfile: stat: %w", err)} | |
| } | |
| res = openFile(f, fi.Size(), firstRead) | |
| return res |
I realize #1126 is still a draft, so no pressure on this. If you do move forward with it, though, this needs to be a defer rather than the two explicit closes: once the open goroutine recovers, a panic inside openFile unwinds through doOpen without reaching either close, and the OnceValue then re-raises on every later call, so Close never gets to the file either. A defer is the only construct that runs on that path. The explicit closes are correct for every error return today, so this is purely about not letting #1126's "a panic fails the read" guarantee come with a leaked fd per failure.
| r: stores.OpenPack(path, packfile.ReaderOptions{RecordDecoder: coldPackDecoder}), | ||
| r: stores.OpenPack(path, packfile.ReaderOptions{ | ||
| RecordDecoder: coldPackDecoder, | ||
| FirstRead: func(int64) int { return coldFirstRead }, |
There was a problem hiding this comment.
Is there a test that fails when a format change pushes a tail past its FirstRead, or when events.pack opens at pubnet density start taking two reads? The trailer is enough: IndexSize + AppDataSize + 76 is the tail, and a store test can compare it against the FirstRead it configures with no new hooks. writeFixturePack at 10,000 ledgers covers this store, and buildColdFixture at 10,000 ledgers and a pinned pubnet-with-margin density covers events.pack (the offsets blob scales with ledgers, not events, so the ledger count is what matters). A fixture can't produce 32-bit widths, so the derived bound still has to cover widths, but this catches the #1077 kind of drift, where the layout grows and nobody re-reads the constant.
There was a problem hiding this comment.
There isn't one for these stores; only index.pack has one (TestIndexPack_IndexCostsAtMostFourBytesPerRecord in #1082). I don't think a fixture test would catch the drift here, though. writeFixturePack's 10,000-ledger tail is 6.5 KB, so a change like #1077's directory entries would pass with about 58 KB to spare. For events.pack, the tail is mostly the record index, which grows with the number of events rather than ledgers. Since an outgrown tail costs one extra read rather than a failure, I'd rather not add a test that can't fail on the change it guards against.
| readBufSize = 1 << 20 // 1 MiB pooled coalesced-read buffer | ||
| speculativeReadSize = 256 * 1024 // 256 KiB tail prefetch on Open | ||
| readBufSize = 1 << 20 // 1 MiB pooled coalesced-read buffer | ||
| defaultFirstRead = 256 << 10 |
There was a problem hiding this comment.
By the top of the stack events.pack is the only pack still on this default, and 256 KiB fits it badly at both ends. Its tail is the offsets blob (40 KB, 13.8 KB after #1085) plus about 4 B per 128-event record, so at pubnet density it is well under 64 KiB and the open reads ~200 KiB nobody uses: about 1.6 ms of volume throughput and half a millisecond of latency per open, the same size as this PR's ledger saving. The record part tracks file size about as well as index.pack's does in #1082, so events.pack could get its own FirstRead of the same shape, max(64 KiB, offsets max + c × fileSize) with c from the bench tree's p99 of tail over file size. That is not a hard bound, but a miss only costs the second read that dense chunks already pay, and the choice then lives next to the events format instead of in the library default.
There was a problem hiding this comment.
I think the tail is bigger than that at pubnet density. Records close at 128 events, about 6 KB compressed, so a pubnet chunk has about 80,000 of them. On chunk 6465 the record index alone is 133 KB and the whole tail is about 158 KB, so 256 KiB covers it in one read. We tried a FirstRead that scales with file size during the design and dropped it: tail over file size is 0.027% on pubnet but 0.0076% on sac-6000, so a rule fitted to pubnet reads about 1.8 MiB per open on sac-6000.
|
also, be sure to rebase! |
|
Follow-up result on my benchmarking I mentioned above that's worth considering! Benchmark: FADV_RANDOM on the packfile fd (my RPCv2 archive node on EBS, real packs, cold cache, 3 seeds, kB fetched from the volume per request) The test runs
Cause: reads over The change is small, in the open path this PR already touches: // packfile/reader.go, doOpen, right after os.Open succeeds
if err := adviseRandom(f); err != nil { ... }
// packfile/fadvise_linux.go (//go:build linux)
func adviseRandom(f *os.File) error {
sc, err := f.SyscallConn(); if err != nil { return err }
var aerr error
if err := sc.Control(func(fd uintptr) { aerr = unix.Fadvise(int(fd), 0, 0, unix.FADV_RANDOM) }); err != nil { return err }
return aerr
}
// packfile/fadvise_other.go (//go:build !linux): return nil
|
|
Thanks for running this. We took it in #1076 rather than here, since it changes how records are read, with three changes from your sketch:
Measured at the top of the stack through the daemon's handlers, on pubnet chunk 6504 on gp3, with the page cache evicted before every request and 900 requests per type:
The full table is in #1076's description. On rebasing: none of the commits #1075 is behind touch its files, so I'll rebase the stack once #1075 lands. |
Why
A small change that pays on EBS deployments. Cold readers are opened per request and never cached, by design, and a request opens every chunk it routes to, so every byte an open reads is paid on every request. On a gp3 volume those bytes come out of the bandwidth quota that queries share. Open read a fixed 256 KiB from the end of every pack, which is four times a ledger pack's tail, and when a dense events.pack's tail was larger it read the whole index region again, repeating the 256 KiB it already had.
The design
Each store knows how its own tail grows, so the store sizes the first read:
ReaderOptions.FirstReadtakes the file size and returns the read size, nil meaning 256 KiB. Ledger packs always hold 10,000 records, so a fixed 64 KiB always covers them. events.pack's tail follows the event count, which is not known until the trailer is read, so it keeps 256 KiB and accepts one extra read on dense chunks. index.pack's tail follows the file size, and its store sets that rule in a later PR. When the first read falls short, Open reads only the missing bytes in front of it, never bytes it already holds.The alternatives considered: one smaller fixed size for every pack, which would make every events.pack open two reads; one file-size formula for every pack, which only fits index.pack, since the other packs' tails do not track their file size; encoding the tail size in the file length or recording it in the catalog, which holds no per-file data.
How it works
The ledger cold reader sets 64 KiB.
doOpenhands the opened file toopenFile, so a test can record every read Open makes, and the new tests pin the exact reads for a tail that fits and one that does not.Measured
This PR has no timing benchmark; its tests check exactly which bytes Open reads. On the bench box's gp3 volume with a cold page cache a single read costs 0.66 ms for 16 KiB, 1.3 ms for 256 KiB and 8 ms for 1 MiB. A ledger open saves about half a millisecond by reading 64 KiB instead of 256 KiB. A dense events.pack open reads 256 KiB less, because the second read no longer repeats what the first one fetched; at the volume's 125 MiB/s that is about 2 ms. On local NVMe the difference is negligible.
🤖 Generated with Claude Code