Skip to content

fix: copy partial Uint8Array views before storing them in IndexedDB - #380

Open
Chandelier-02 wants to merge 1 commit into
ipfs:mainfrom
Chandelier-02:fix/idb-copy-partial-views
Open

Chandelier-02 wants to merge 1 commit into
ipfs:mainfrom
Chandelier-02:fix/idb-copy-partial-views

Conversation

@Chandelier-02

Copy link
Copy Markdown

IndexedDB stores values by structured clone. When a Uint8Array is a view onto part of a larger buffer, the structured clone copies that view's whole backing ArrayBuffer, not just the bytes in view. The block is then stored at the size of its backing buffer, and every read returns that whole buffer too.

IDBBlockstore.put hands the Uint8Array it is given straight to IndexedDB, and Helia often passes views:

  • bitswap: each block received is a subarray of the decoded message, which holds every block the peer sent in that round;
  • unixfs.addBytes: each leaf is sliced out of the whole input buffer, so importing an N MiB file stores about N² MiB;
  • byte streams: chunks of at least the leaf size give leaves that are views into those chunks.

Measured in Chromium:

Case Stored Read time per value
20 × 1 MiB stored as views of one 20 MiB buffer 419 MB 22 ms
The same 20 × 1 MiB stored as copies 21 MB 1.4 ms
Case Stored / data
64 small (30 KiB) blocks fetched together over bitswap 64×
A 10 MiB UnixFS file fetched from a peer 2.8×

This change copies the value before storing it whenever it covers only part of its buffer. A value that already owns its whole buffer is stored as before, with no copy. The copy uses new Uint8Array(buf) rather than buf.slice(), because on a Buffer (including the browser polyfill) slice returns another view.

For comparison, datastore-level does not have this problem in Node.js. It passes the Uint8Array to classic-level, whose native binding reads the view through napi_get_typedarray_info (a data pointer already adjusted by byteOffset, plus the view's length) and memcpys exactly those bytes into LevelDB. The backing buffer never reaches the database, so a view and a copy are stored identically. (In the browser, level resolves to browser-level, which is also backed by IndexedDB and puts the view directly, so that backend has the same behaviour; it lives outside this repo.)

Blocks already stored this way stay bloated until they are rewritten; this change only affects new puts.

The added test fails without the fix: the value read back still has byteOffset 100 and a 1 MiB backing buffer.

🤖 Generated with Claude Code

IndexedDB stores values by structured clone, which copies the whole
ArrayBuffer behind a Uint8Array view rather than just the bytes in view.
Copy a value that covers only part of its buffer before putting it so
the block is stored (and read back) at its own size.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant