[herber]

Note

Medium Risk
Changes core upload paths across S3, Azure, and GCS plus new client APIs; failures during multipart/block uploads could leave partial data unless abort paths run, but existing small-object behavior is largely preserved.

Overview
Adds streaming I/O so objects larger than memory can be transferred without buffering the full payload in process memory.

The Rust object-store backends introduce shared ChunkedUpload logic (8 MB single-request threshold, 16 MB parts). S3 uses multipart upload with abort on failure; Azure stages blocks and commits a block list; GCS uses streamed upload when over the threshold. Small objects still use the existing single-request paths.

The Go client adds GetObjectStream (returns metadata plus an io.ReadCloser the caller must close) and PutObjectFromReader (context-aware PUT from an io.Reader, including unknown size). Header metadata parsing is centralized in objectMetadataFromHeaders, and error responses are capped when reading bodies.

^{Reviewed by Cursor Bugbot for commit 12e43a6. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 12e43a6. Configure here.}

[cursor]

Azure leaves staged blocks behind

Medium Severity

put_object_blocks stages blocks then commits them, but on a failed put_block or put_block_list it returns without discarding uncommitted blocks. Unlike the new S3 path, which aborts the multipart upload on failure, Azure can keep that staged data around for days and bill for it after a failed large upload.

 

^{Reviewed by Cursor Bugbot for commit 12e43a6. Configure here.}

[cursor]

GCS streams via simple upload

High Severity

When an object exceeds SINGLE_REQUEST_THRESHOLD, GCS still uploads with UploadType::Simple and a stream of unknown length, without setting Media.content_length. S3 and Azure switch to multipart/block APIs for that case. Simple GCS uploads expect a known length (or a fragile chunked POST), so larger-than-memory GCS uploads are likely to fail or be unreliable.

 

^{Reviewed by Cursor Bugbot for commit 12e43a6. Configure here.}

[cursor]

Default client timeout blocks streaming

High Severity

GetObjectStream and PutObjectFromReader are meant for larger-than-memory transfers, but they use the shared http.Client from NewClient, which sets a 30s Timeout covering the full body. Long uploads or downloads can fail mid-stream even after a successful Do, which undermines the new streaming APIs when callers use the default client.

Additional Locations (1)

• clients/go/client.go#L396-L443

 

^{Reviewed by Cursor Bugbot for commit 12e43a6. Configure here.}