Skip to content

packfile: read only the missing tail bytes at open - #1075

Merged
tamirms merged 1 commit into
feature/full-historyfrom
tamirms/packfile-tail-reads
Oct 21, 2026
Merged

tamirms merged 1 commit into
feature/full-historyfrom
tamirms/packfile-tail-reads

Conversation

@tamirms

@tamirms tamirms commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Why

A small change that pays on EBS deployments. Cold readers are opened per request and never cached, by design, and a request opens every chunk it routes to, so every byte an open reads is paid on every request. On a gp3 volume those bytes come out of the bandwidth quota that queries share. Open read a fixed 256 KiB from the end of every pack, which is four times a ledger pack's tail, and when a dense events.pack's tail was larger it read the whole index region again, repeating the 256 KiB it already had.

The design

Each store knows how its own tail grows, so the store sizes the first read: ReaderOptions.FirstRead takes the file size and returns the read size, nil meaning 256 KiB. Ledger packs always hold 10,000 records, so a fixed 64 KiB always covers them. events.pack's tail follows the event count, which is not known until the trailer is read, so it keeps 256 KiB and accepts one extra read on dense chunks. index.pack's tail follows the file size, and its store sets that rule in a later PR. When the first read falls short, Open reads only the missing bytes in front of it, never bytes it already holds.

The alternatives considered: one smaller fixed size for every pack, which would make every events.pack open two reads; one file-size formula for every pack, which only fits index.pack, since the other packs' tails do not track their file size; encoding the tail size in the file length or recording it in the catalog, which holds no per-file data.

How it works

The ledger cold reader sets 64 KiB. doOpen hands the opened file to openFile, so a test can record every read Open makes, and the new tests pin the exact reads for a tail that fits and one that does not.

Measured

This PR has no timing benchmark; its tests check exactly which bytes Open reads. On the bench box's gp3 volume with a cold page cache a single read costs 0.66 ms for 16 KiB, 1.3 ms for 256 KiB and 8 ms for 1 MiB. A ledger open saves about half a millisecond by reading 64 KiB instead of 256 KiB. A dense events.pack open reads 256 KiB less, because the second read no longer repeats what the first one fetched; at the volume's 125 MiB/s that is about 2 ms. On local NVMe the difference is negligible.

🤖 Generated with Claude Code

Copilot AI balanced review requested due to automatic review settings October 5, 2026 10:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation is localized, preserves default behavior, and thoroughly tests boundary and fallback cases.

Review effort: Balanced
Findings: None

What changed in this PR

Optimizes packfile opening by reading only missing tail bytes and tuning ledger cold-pack reads.

Changes:

  • Adds configurable first-read sizing.
  • Avoids rereading already-fetched tail data.
  • Uses 64 KiB first reads for ledger cold packs and adds coverage.
File Description
design-docs/​packfile-library.md Documents first-read behavior.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​ledger/​cold_reader.go Configures 64 KiB ledger tail reads.
cmd/​stellar-rpc/​internal/​rpcv2/​packfile/​tail_test.go Tests read boundaries and fallback behavior.
cmd/​stellar-rpc/​internal/​rpcv2/​packfile/​reader.go Implements configurable, non-overlapping tail reads.
cmd/​stellar-rpc/​internal/​rpcv2/​packfile/​packfile.go Updates validation documentation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@tamirms
tamirms added this pull request to stack #1081 October 5, 2026 10:56
@tamirms
tamirms removed this pull request from stack #1081 October 5, 2026 16:19
@tamirms
tamirms added this pull request to stack #1089 October 5, 2026 16:24
@tamirms tamirms added this to the platform sprint 76 milestone Oct 6, 2026
@tamirms
tamirms removed this pull request from stack #1089 October 8, 2026 08:18
Copilot AI balanced review requested due to automatic review settings October 8, 2026 08:18
@tamirms
tamirms force-pushed the tamirms/packfile-tail-reads branch from a6444df to b1b0904 Compare October 8, 2026 08:18
@tamirms
tamirms added this pull request to stack #1102 October 8, 2026 08:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation matches the design and covers boundary and fallback behavior with focused tests.

0 open findings

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

@tamirms
tamirms force-pushed the tamirms/packfile-tail-reads branch from b1b0904 to 4cf054b Compare October 8, 2026 08:34
Copilot AI balanced review requested due to automatic review settings October 8, 2026 08:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation matches the described optimization and covers its boundary cases with focused tests.

0 open findings

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Base automatically changed from tamirms/routed-fingerprint to feature/full-history October 8, 2026 08:36
@tamirms
tamirms force-pushed the tamirms/packfile-tail-reads branch from 4cf054b to 2db23ff Compare October 8, 2026 08:36
Copilot AI balanced review requested due to automatic review settings October 9, 2026 06:02
@tamirms
tamirms force-pushed the tamirms/packfile-tail-reads branch from 2db23ff to 05619c3 Compare October 9, 2026 06:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation is focused, preserves fallback correctness, and adequately tests the new read behavior.

0 open findings

🧠 Review effort: Balanced

@tamirms
tamirms requested a review from Shaptic October 12, 2026 20:34
Open reads the last 256 KiB of every pack. When the index, app data and
trailer are larger, it reads the whole index region again, up to 256 KiB
more than it needs; events.pack's index reaches 0.8 to 1 MB on large
chunks. For a ledger pack 256 KiB is too much: its tail is under 42 KB
(16 to 28 KB on real chunks), and a request opens every chunk it routes
to.

ReaderOptions.FirstRead now sizes Open's first read from the file size,
so a store whose tail grows with the file can follow it; nil keeps
256 KiB. When the first read falls short of the tail, Open reads only
the missing bytes in front of it. The ledger cold reader sets 64 KiB,
which always covers its tail.

doOpen now hands the opened file to openFile, so a test can record every
read Open makes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KKssDnVqUbxVZqpxPN3FpP
Copilot AI balanced review requested due to automatic review settings October 15, 2026 18:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused changes preserve validation and cleanup, with regression tests covering read boundaries and fallback behavior.

0 open findings

🧠 Review effort: Balanced

@cjonas9 cjonas9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this LGTM! I have a few comments and thoughts, but none are worth blocking over.

Heads up: I'm benchmarking this PR on my own RPCv2 node (EBS) to see if adding FADV_RANDOM on the packfile fd at open gives a substantial drop in bytes read per request.

Why: this PR's win is fewer bytes per open on EBS. The record reads after the open use the same fd, and kernel readahead can pull in bytes nobody asked for on top of the coalescing #1076 already does. If that overshoot is as big as the tail saving, it's a bigger lever in the same open path and worth knowing before we size this win.

Numbers across three seeds in a couple of hours.


// coldFirstRead covers a cold pack's whole tail: a chunk's index is under
// 42 KB at any ledger size, and the app data and trailer under 100 bytes.
const coldFirstRead = 64 << 10

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imo, I think there's a good case for deriving this instead of hard-coding it. Neither reason is crucial, but I think it's a fairly low-cost change:

  • this number could drift over time. We have a lot of buffer now between ~40KB and 64KB, packfile: decode only the index groups a read touches #1077's 17-byte directory entries make it 41,823 B. If packfile exports those sizes, coldFirstRead becomes a constant expression (rounded up to a page) that moves with the format, and nobody has to re-check the 64 KiB claim when the index layout changes
  • it trims 16KB per open. My back-of-the-envelope math makes the improvement seem fairly modest, but a bonus nonetheless

Thoughts?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working through the numbers. I'd keep 64 KiB. Deriving it from today's worst case (41,823 B, so 44 KiB rounded to a page) saves about 20 KiB inside the same single read. The headroom is there on purpose: the ledger layout planned for LCM V3 puts typical tails at about 45 to 53 KB, past what a derived value would allow. And a tail that outgrows the first read costs one extra read, not a failure. Deriving it would also mean exporting a worst-case size formula from packfile for this one caller.

Comment on lines 222 to +231
fi, err := f.Stat()
if err != nil {
_ = f.Close()
return openResult{err: fmt.Errorf("packfile: stat: %w", err)}
}
fileSize := fi.Size()
res := openFile(f, fi.Size(), firstRead)
if res.err != nil {
_ = f.Close()
}
return res

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
fi, err := f.Stat()
if err != nil {
_ = f.Close()
return openResult{err: fmt.Errorf("packfile: stat: %w", err)}
}
fileSize := fi.Size()
res := openFile(f, fi.Size(), firstRead)
if res.err != nil {
_ = f.Close()
}
return res
var res openResult
// Closes f unless res took it; a panic in openFile leaves res empty, so it closes then too.
defer func() {
if res.file == nil {
_ = f.Close()
}
}()
fi, err := f.Stat()
if err != nil {
return openResult{err: fmt.Errorf("packfile: stat: %w", err)}
}
res = openFile(f, fi.Size(), firstRead)
return res

I realize #1126 is still a draft, so no pressure on this. If you do move forward with it, though, this needs to be a defer rather than the two explicit closes: once the open goroutine recovers, a panic inside openFile unwinds through doOpen without reaching either close, and the OnceValue then re-raises on every later call, so Close never gets to the file either. A defer is the only construct that runs on that path. The explicit closes are correct for every error return today, so this is purely about not letting #1126's "a panic fails the read" guarantee come with a leaked fd per failure.

@tamirms tamirms Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. Your defer is in #1126 as 9c5401e: doOpen now closes the file on every path that doesn't hand it to the Reader, a panic in openFile included.

r: stores.OpenPack(path, packfile.ReaderOptions{RecordDecoder: coldPackDecoder}),
r: stores.OpenPack(path, packfile.ReaderOptions{
RecordDecoder: coldPackDecoder,
FirstRead: func(int64) int { return coldFirstRead },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a test that fails when a format change pushes a tail past its FirstRead, or when events.pack opens at pubnet density start taking two reads? The trailer is enough: IndexSize + AppDataSize + 76 is the tail, and a store test can compare it against the FirstRead it configures with no new hooks. writeFixturePack at 10,000 ledgers covers this store, and buildColdFixture at 10,000 ledgers and a pinned pubnet-with-margin density covers events.pack (the offsets blob scales with ledgers, not events, so the ledger count is what matters). A fixture can't produce 32-bit widths, so the derived bound still has to cover widths, but this catches the #1077 kind of drift, where the layout grows and nobody re-reads the constant.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There isn't one for these stores; only index.pack has one (TestIndexPack_IndexCostsAtMostFourBytesPerRecord in #1082). I don't think a fixture test would catch the drift here, though. writeFixturePack's 10,000-ledger tail is 6.5 KB, so a change like #1077's directory entries would pass with about 58 KB to spare. For events.pack, the tail is mostly the record index, which grows with the number of events rather than ledgers. Since an outgrown tail costs one extra read rather than a failure, I'd rather not add a test that can't fail on the change it guards against.

readBufSize = 1 << 20 // 1 MiB pooled coalesced-read buffer
speculativeReadSize = 256 * 1024 // 256 KiB tail prefetch on Open
readBufSize = 1 << 20 // 1 MiB pooled coalesced-read buffer
defaultFirstRead = 256 << 10

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the top of the stack events.pack is the only pack still on this default, and 256 KiB fits it badly at both ends. Its tail is the offsets blob (40 KB, 13.8 KB after #1085) plus about 4 B per 128-event record, so at pubnet density it is well under 64 KiB and the open reads ~200 KiB nobody uses: about 1.6 ms of volume throughput and half a millisecond of latency per open, the same size as this PR's ledger saving. The record part tracks file size about as well as index.pack's does in #1082, so events.pack could get its own FirstRead of the same shape, max(64 KiB, offsets max + c × fileSize) with c from the bench tree's p99 of tail over file size. That is not a hard bound, but a miss only costs the second read that dense chunks already pay, and the choice then lives next to the events format instead of in the library default.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the tail is bigger than that at pubnet density. Records close at 128 events, about 6 KB compressed, so a pubnet chunk has about 80,000 of them. On chunk 6465 the record index alone is 133 KB and the whole tail is about 158 KB, so 256 KiB covers it in one read. We tried a FirstRead that scales with file size during the design and dropped it: tail over file size is 0.027% on pubnet but 0.0076% on sac-6000, so a rule fitted to pubnet reads about 1.8 MiB per open on sac-6000.

@cjonas9

cjonas9 commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

also, be sure to rebase!

@cjonas9

cjonas9 commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Follow-up result on my benchmarking I mentioned above that's worth considering!

Benchmark: FADV_RANDOM on the packfile fd (my RPCv2 archive node on EBS, real packs, cold cache, 3 seeds, kB fetched from the volume per request)

The test runs bench query (the daemon's read handlers) against my archive node's packs with the page cache evicted, and counts the bytes the volume serves per request, for three builds on identical seeded requests: feature/full-history tip, tip + this PR, and tip + this PR + FADV_RANDOM on the packfile fd at open. The win is volume throughput/IOPS headroom, same currency as this PR.

request base this PR (vs base) PR + FADV_RANDOM (vs PR)
getLedgers (limit 10) 1300 kB 1118 kB (−14%) 907 kB (−19%)
getTransactions (limit 200) 2057 kB 1703 kB (−17%) 1392 kB (−18%)
queryEvents (limit 10) 1543 kB 1554 kB (0%) 1368 kB (−12%)
txhash lookup 628 kB 523 kB (−17%) 535 kB (0%, index is mmap'd)

Cause: reads over read_ahead_kb (128 KB) go through the kernel's readahead path and the last window overshoots the request, so each coalesced record read fetches ~one window nobody uses.

The change is small, in the open path this PR already touches:

// packfile/reader.go, doOpen, right after os.Open succeeds
if err := adviseRandom(f); err != nil { ... }

// packfile/fadvise_linux.go (//go:build linux)
func adviseRandom(f *os.File) error {
    sc, err := f.SyscallConn(); if err != nil { return err }
    var aerr error
    if err := sc.Control(func(fd uintptr) { aerr = unix.Fadvise(int(fd), 0, 0, unix.FADV_RANDOM) }); err != nil { return err }
    return aerr
}

// packfile/fadvise_other.go (//go:build !linux): return nil

golang.org/x/sys/unix is already in go.mod. Happy to see it folded in here since it's the same open path, or as a follow-up if you'd rather keep this PR to the tail read.

@tamirms

tamirms commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for running this. We took it in #1076 rather than here, since it changes how records are read, with three changes from your sketch:

  • ReadRange prefetches its next read with POSIX_FADV_WILLNEED from its second read on, never past the requested range. getTransactions and unfiltered queryEvents ask for a wide range and usually stop inside the first read, so prefetching from the first read would fetch up to 256 KiB they never use.
  • With readahead off, the kernel serves a read larger than one I/O as 256 KiB pieces, one after another. A read that spans more than one I/O now hints all of its pieces first. That covers large ledgers, and events.pack's 256 KiB tail read when it ends mid-page and touches 65 pages. Without it, filtered queryEvents got 6.5% slower at p99.
  • Merged reads count pages, from your pageSpan suggestion on packfile: merge adjacent records into reads of up to 256 KiB #1076.

Measured at the top of the stack through the daemon's handlers, on pubnet chunk 6504 on gp3, with the page cache evicted before every request and 900 requests per type:

  • getLedgers of 10 ledgers: p50 from 34.8 to 32.4 ms, with 11% fewer bytes.
  • queryEvents for one contract: p50 from 4.39 to 3.83 ms.
  • queryEvents with no filter: p50 from 4.20 to 3.45 ms, with 14% fewer bytes.
  • getTransactions and getTransaction by hash: flat at p50, and flat or better at p99.

The full table is in #1076's description. On rebasing: none of the commits #1075 is behind touch its files, so I'll rebase the stack once #1075 lands.

@tamirms
tamirms merged commit 8f09e43 into main Oct 21, 2026
17 checks passed
@tamirms
tamirms deleted the tamirms/packfile-tail-reads branch October 21, 2026 09:31
@github-project-automation github-project-automation Bot moved this from Needs Review to Done in Platform Scrum Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants