Skip to content

[python] Route descriptor-backed BLOB reads through the table FileIO. - #9608

Open
Stephen0421 wants to merge 1 commit into
apache:masterfrom
Stephen0421:pypaimon-blob-pr1c-read-path
Open

[python] Route descriptor-backed BLOB reads through the table FileIO.#9608
Stephen0421 wants to merge 1 commit into
apache:masterfrom
Stephen0421:pypaimon-blob-pr1c-read-path

Conversation

@Stephen0421

Copy link
Copy Markdown
Contributor

Purpose

Route descriptor-backed BLOB reads through the table FileIO, and resolve known descriptor columns with from_descriptor_bytes instead of the v2-only from_bytes() heuristic.

  • UriReaderFactory.from_file_io matches Java fromFileIO: HTTP(S) stays on HttpUriReader; every other URI reuses the table FileIO (REST tokens). Non-HTTP readers are not LRU-cached, so the factory does not pin FileIO. After pickle, a ResolvingFileIO table still works once uri_reader_factory has been materialized.
  • Convert reader and OffsetRow.get_blob() parse descriptor-field bytes with from_descriptor_bytes (v1, trailing padding, and VideoFrameDescriptor via serde).
  • After materializing payload (blob-as-descriptor=false), descriptor field indices are cleared so row-level get_blob() does not re-parse payload as a descriptor. Reader wrappers refresh blob-view lookup only and do not overwrite those indices.
  • Apply LIMIT before inline convert on merge splits, keep RowKind through the row/batch/row adapter, and keep sequence.field on blob-view prescan so overlapping-file merge can still compare.
  • read_blobs_concurrent coalesces exact BlobRef + FileUriReader only; subclasses keep new_input_stream().

Out of scope: when blob-as-descriptor=true, Stage 1 still rewrites BlobViewStruct to serialized descriptor bytes. Keeping the struct for that mode is a follow-up.

Tests

  • BlobTest: v1/trailing-padding/VideoFrameDescriptor via from_descriptor_bytes; table FileIO on convert/get_blob/deferred resolve; LIMIT before convert; RowKind through adapters; exact-BlobRef coalesce; merge blob-view prescan keeps sequence.field; _refresh_blob_view_lookup does not clobber descriptor indices
  • DedicatedFormatWriterTest.test_blob_view_predicate_and_limit_resolves_filtered_row
  • DedicatedFormatWriterTest.test_blob_view_raw_split_predicate_and_limit_resolves_filtered_row
  • UriReaderFactoryTest: from_file_io reuses FileIO for non-HTTP, does not pin FileIO, and pickles with FileIO in the graph
  • ResolvingFileIOTest: pickle FileIO and table after uri_reader_factory is materialized
  • RESTTokenFileIOTest.test_close_only_closes_instance_uri_reader_factory

Reuse UriReaderFactory.from_file_io for non-HTTP URIs, resolve known
descriptor fields with from_descriptor_bytes (including v1 writes), and
keep RowKind/LIMIT stable around inline convert. Do not pin FileIO in the
from_file_io cache, and keep ResolvingFileIO tables pickleable after the
factory is materialized.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant