[python] Route descriptor-backed BLOB reads through the table FileIO. - #9608
Open
Stephen0421 wants to merge 1 commit into
Open
[python] Route descriptor-backed BLOB reads through the table FileIO.#9608Stephen0421 wants to merge 1 commit into
Stephen0421 wants to merge 1 commit into
Conversation
Reuse UriReaderFactory.from_file_io for non-HTTP URIs, resolve known descriptor fields with from_descriptor_bytes (including v1 writes), and keep RowKind/LIMIT stable around inline convert. Do not pin FileIO in the from_file_io cache, and keep ResolvingFileIO tables pickleable after the factory is materialized.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Route descriptor-backed BLOB reads through the table FileIO, and resolve known descriptor columns with
from_descriptor_bytesinstead of the v2-onlyfrom_bytes()heuristic.UriReaderFactory.from_file_iomatches JavafromFileIO: HTTP(S) stays onHttpUriReader; every other URI reuses the table FileIO (REST tokens). Non-HTTP readers are not LRU-cached, so the factory does not pin FileIO. After pickle, a ResolvingFileIO table still works onceuri_reader_factoryhas been materialized.OffsetRow.get_blob()parse descriptor-field bytes withfrom_descriptor_bytes(v1, trailing padding, andVideoFrameDescriptorvia serde).blob-as-descriptor=false), descriptor field indices are cleared so row-levelget_blob()does not re-parse payload as a descriptor. Reader wrappers refresh blob-view lookup only and do not overwrite those indices.sequence.fieldon blob-view prescan so overlapping-file merge can still compare.read_blobs_concurrentcoalesces exactBlobRef+FileUriReaderonly; subclasses keepnew_input_stream().Out of scope: when
blob-as-descriptor=true, Stage 1 still rewritesBlobViewStructto serialized descriptor bytes. Keeping the struct for that mode is a follow-up.Tests
BlobTest: v1/trailing-padding/VideoFrameDescriptorviafrom_descriptor_bytes; table FileIO on convert/get_blob/deferred resolve; LIMIT before convert; RowKind through adapters; exact-BlobRefcoalesce; merge blob-view prescan keepssequence.field;_refresh_blob_view_lookupdoes not clobber descriptor indicesDedicatedFormatWriterTest.test_blob_view_predicate_and_limit_resolves_filtered_rowDedicatedFormatWriterTest.test_blob_view_raw_split_predicate_and_limit_resolves_filtered_rowUriReaderFactoryTest:from_file_ioreuses FileIO for non-HTTP, does not pin FileIO, and pickles with FileIO in the graphResolvingFileIOTest: pickle FileIO and table afteruri_reader_factoryis materializedRESTTokenFileIOTest.test_close_only_closes_instance_uri_reader_factory