Skip to content

txhash: write the .bin in streamhash's sorted-file format - #1156

Draft
tamirms wants to merge 1 commit into
tamirms/cold-read-panics-to-errorsfrom
tamirms/txhash-ga-format
Draft

tamirms wants to merge 1 commit into
tamirms/cold-read-panics-to-errorsfrom
tamirms/txhash-ga-format

Conversation

@tamirms

@tamirms tamirms commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

A later change in this stack, "txhash: build the window index from the sorted .bin files, between ledgers", builds the tx-hash window index with streamhash's BuildFromSortedFiles, which reads streamhash's sorted-file format. This changes the .bin files to that format now, with rpc's own writer and reader, so the files a GA node writes stay the same when that build lands, and nodes keep their data.

  • Layout: a header with streamhash's magic, version, payload size, record count and a metadata blob (a version byte, the chunk and the index secret), padded to 4 KiB, then 19-byte records: the 16-byte routing key and a 3-byte little-endian payload. The old layout had a 32-byte header and 20-byte records holding the absolute ledger.
  • Payload: a record stores its ledger minus the first ledger of the chunk's index window. That base stays fixed for the window's life, so the payload is final when the .bin is written. The backfill derives the base beside the chunk's secret and passes it in ingest.Config, and the writer refuses a ledger below it.
  • Build: today's merge reads the new layout and copies each stored payload instead of subtracting. The index records the window's first ledger in the field that held the coverage's first ledger, so the reader and the index metadata layout don't change.

The getTransaction design describes the new layout.

Tests: the .bin layout test, the backfill format pin and the shared test decoder check the new bytes, and a new test covers the writer refusing a ledger below the base. Writing the same chunks and building their window index with this change and with the stack's final code gives byte-identical files.

Everyone upgrading to GA ingests from scratch, so there is no migration.

🤖 Generated with Claude Code

@tamirms
tamirms added this pull request to stack #1157 October 27, 2026 10:21
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch from bbeefcc to 5471555 Compare October 28, 2026 08:05
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch from 5471555 to 48a5564 Compare November 4, 2026 10:07
@tamirms
tamirms removed this pull request from stack #1157 November 4, 2026 10:20
@tamirms
tamirms added this pull request to stack #1184 November 4, 2026 10:21
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch 2 times, most recently from ba0cae0 to 0a815ef Compare November 4, 2026 19:12
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch 2 times, most recently from 599f281 to ebcdedd Compare November 12, 2026 09:00
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch 3 times, most recently from 9d00760 to e6fb1ec Compare November 12, 2026 13:32
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch 2 times, most recently from d791781 to f9e1f89 Compare November 13, 2026 07:25
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch from f9e1f89 to da9c6db Compare November 13, 2026 07:50
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch from da9c6db to f2e76de Compare November 13, 2026 09:55
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch from f2e76de to fad0ea5 Compare November 13, 2026 11:38
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch from fad0ea5 to 357547c Compare November 13, 2026 12:34
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch 3 times, most recently from c7ba01b to 281bd33 Compare November 14, 2026 08:41
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch from 281bd33 to 0dea352 Compare November 16, 2026 07:51
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch from 0dea352 to 38387a7 Compare November 16, 2026 10:29
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch from 38387a7 to 5203b42 Compare November 16, 2026 13:17
A chunk's .bin now has the bytes of a streamhash sorted file: the STSF
header with the payload size, the entry count and a metadata blob holding
the chunk ID and the index secret, padded to 4096 bytes, then 19-byte
records whose 3-byte payload is the ledger minus the first ledger of the
chunk's index window. The window build reads the new layout and stores
each payload as it is, and records the window's first ledger as the
index's MinLedger, so the reader still adds MinLedger to a payload.

The bytes match what the index build from sorted files writes, so it can
replace this build without changing anything on disk. The index metadata
layout does not change. Nodes re-ingest the tx-hash tier from scratch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tamirms
tamirms force-pushed the tamirms/txhash-ga-format branch from 5203b42 to db5fc85 Compare November 16, 2026 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant