Skip to content

GH-51156: [C++] Raise CapacityError instead of truncating binary values over 2 GiB - #51158

Open
ngoldbaum wants to merge 1 commit into
apache:mainfrom
ngoldbaum:fix-truncation
Open

GH-51156: [C++] Raise CapacityError instead of truncating binary values over 2 GiB#51158
ngoldbaum wants to merge 1 commit into
apache:mainfrom
ngoldbaum:fix-truncation

Conversation

@ngoldbaum

Copy link
Copy Markdown

Rationale for this change

BaseBinaryBuilder::Append(std::string_view), ExtendCurrent(std::string_view) and ChunkedBinaryBuilder::Append(std::string_view) cast the value size to int32_t before validating it, so a value of 2 GiB or more is stored as an empty or truncated value. See #51156.

What changes are included in this PR?

The three overloads validate the 64-bit size before narrowing it, returning the same CapacityError that ValidateOverflow produces for the other paths.

Are these changes tested?

Yes, two LARGE_MEMORY_TEST cases in array_binary_test.cc. They fail without the fix.

Are there any user-facing changes?

BinaryBuilder, StringBuilder and ChunkedBinaryBuilder now return CapacityError for a string_view value of 2 GiB or more instead of storing wrong data.

This PR contains a "Critical Fix". The builders produced incorrect data for such values.

AI disclosure: I used an AI model to think about and iterate on the fix. It also originally identified the bug.

@ngoldbaum
ngoldbaum requested a review from pitrou as a code owner September 3, 2026 20:28
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #51156 has been automatically assigned in GitHub to PR creator.

@andishgar

Copy link
Copy Markdown
Contributor

@pitrou, I have already solved this problem here to enable BinaryViewBuilder to accept more than 2GB of data. It has been long-awaited in the review. :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants