[common] Default missing variant shredding fields to the unshredded index - #9616
Open
LuciferYang wants to merge 1 commit into
Open
[common] Default missing variant shredding fields to the unshredded index#9616LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
…ndex BaseVariantReader.RowReader's constructor unboxed schema.objectSchemaMap.get(name) into an int[]. The map only holds fields the file actually shredded, so reading a struct whose fields are not all shredded threw a NullPointerException while building the reader. Use getOrDefault(name, -1), the convention the class already documents and the value its unshredded merge branch waits for.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
close #9615
BaseVariantReader.RowReader's constructor unboxed a map lookup straight into anint:objectSchemaMapholds only the fields the file actually shredded, so reading a struct whose fields are not all shredded threw aNullPointerExceptionwhile the reader was being built, taking the whole scan with it. Shredding is inferred per data file by default, so the same query can work on one file and fail on the next.getOrDefault(name, -1)is all it takes, because -1 is the convention the rest of the class already speaks. The field's own comment says "or -1 if it doesn't exist in objecttyped_value",VariantSchemadocuments the same thing for its indices, andreadFromTypedcarries the branch that reads such a field out of the untyped value:Since the constructor could never produce -1, that branch and the
needUnshreddedObjectflag guarding it were both dead code. This makes them reachable, which is the behavior the class was written for.I checked the other lookups in
org.apache.paimon.data.variantwhile I was there: they all take the result as anIntegerand null-check it, or go throughcontainsKey, so this was the only site.Tests
BaseVariantReaderTest.testRowReaderWithUnshreddedTargetFieldbuilds a shredding schema covering only fielda, asks forstruct<a int, b string>, and reads a row back.bis asserted to come through as"hello", which is only reachable through the unshredded branch, so the test covers the revived path rather than just the absence of the crash. A second case reads{"a": 27}, where the file has no untyped value at all, and expectsbto be null.Against the unfixed reader the test fails with a
NullPointerExceptionout ofBaseVariantReader.create.mvn -pl paimon-common -Dtest=BaseVariantReaderTest teston JDK 8: 1 test, 0 failures.spotless:checkandcheckstyle:checkon paimon-common are clean.