Repository navigation
feat: add optional streaming object storage capability - #32
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
cmilesio
marked this pull request as ready for review
October 18, 2026 07:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add an optional
storagecore.ObjectStoragecapability andstorage.Objects(disk)discovery helper. The S3 driver supports paged object iteration, streaming reads and writes, content metadata, version-pinned reads, and ETag conditions. Opaque object keys remain relative to the disk prefix and retain dot segments and repeated slashes.Keep existing
Storage, bufferedGet/Put, path semantics, other drivers, and minimum Go versions unchanged. Drivers without streaming support returnErrUnsupportedinstead of buffering an entire object as a fallback. Stream callers close readers and provide an explicit size for writes. Bucket provisioning is opt-in throughs3storage.PrepareBucketContext, using existing driver configuration and refusing to create after a permission failure.Why
Framework application backups need bounded object copies that preserve metadata and detect source changes. These provider operations belong in the storage library rather than a second S3 implementation in the backup package. GoForj PR #246 consumes this capability while retaining archive format and checksum ownership.
Unit and race tests cover streamed round trips, opaque keys, metadata, conditional reads, cancellation, callback and provider failures, invalid pagination and prefix escapes, and missing versus forbidden buckets. Root and S3 module tests also passed with
GOWORK=offagainst the published core commit.An independent reviewer agent examined the change and reproduced three issues: XML-sensitive object keys, non-seekable HTTP uploads, and cancellation during SDK prehashing. Follow-up fixes request URL-encoded listings and decode keys exactly once, preserve explicit payload-signing policy while supporting opted-in HTTP streams, and stop further source reads on cancellation. Endpoint handling covers normalized schemes and AWS endpoint overrides. The independent re-review reported no remaining actionable findings.
Protocol tests exercise actual SDK requests against local HTTP and HTTPS servers, including retained HTTPS streaming checksums. The S3 race suite and vet pass, and root/core tests pass with Go 1.24.4 and GOWORK=off. Object operations and bucket preparation have 100% local statement coverage. This review did not exercise a live AWS account.