Repository navigation
fix: copy partial Uint8Array views before storing them in IndexedDB - #380
Open
Chandelier-02 wants to merge 1 commit into
Open
Chandelier-02 wants to merge 1 commit into
Chandelier-02 wants to merge 1 commit into
Conversation
IndexedDB stores values by structured clone, which copies the whole ArrayBuffer behind a Uint8Array view rather than just the bytes in view. Copy a value that covers only part of its buffer before putting it so the block is stored (and read back) at its own size. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
IndexedDB stores values by structured clone. When a
Uint8Arrayis a view onto part of a larger buffer, the structured clone copies that view's whole backingArrayBuffer, not just the bytes in view. The block is then stored at the size of its backing buffer, and every read returns that whole buffer too.IDBBlockstore.puthands theUint8Arrayit is given straight to IndexedDB, and Helia often passes views:subarrayof the decoded message, which holds every block the peer sent in that round;unixfs.addBytes: each leaf is sliced out of the whole input buffer, so importing an N MiB file stores about N² MiB;Measured in Chromium:
This change copies the value before storing it whenever it covers only part of its buffer. A value that already owns its whole buffer is stored as before, with no copy. The copy uses
new Uint8Array(buf)rather thanbuf.slice(), because on aBuffer(including the browser polyfill)slicereturns another view.For comparison,
datastore-leveldoes not have this problem in Node.js. It passes theUint8Arraytoclassic-level, whose native binding reads the view throughnapi_get_typedarray_info(a data pointer already adjusted bybyteOffset, plus the view's length) andmemcpys exactly those bytes into LevelDB. The backing buffer never reaches the database, so a view and a copy are stored identically. (In the browser,levelresolves tobrowser-level, which is also backed by IndexedDB and puts the view directly, so that backend has the same behaviour; it lives outside this repo.)Blocks already stored this way stay bloated until they are rewritten; this change only affects new puts.
The added test fails without the fix: the value read back still has
byteOffset100 and a 1 MiB backing buffer.🤖 Generated with Claude Code