Repository navigation
Add SQL platform support for ParquetSource - #793
JunWang222 wants to merge 9 commits into
Conversation
…-independent-platforms-clean # Conflicts: # wayang-platforms/wayang-jdbc-template/src/main/java/org/apache/wayang/jdbc/execution/JdbcExecutor.java # wayang-platforms/wayang-trino/src/main/resources/wayang-trino-defaults.properties
…m/apache/wayang into feature/parquet-sql-independent-platforms-clean # Conflicts: # wayang-platforms/wayang-bigquery/src/main/resources/wayang-bigquery-defaults.properties # wayang-platforms/wayang-jdbc-template/src/main/java/org/apache/wayang/jdbc/execution/JdbcExecutor.java # wayang-platforms/wayang-presto/src/main/resources/wayang-presto-defaults.properties
| final Tuple<String, String> right = this.keyDescriptor1.getSqlImplementation(); | ||
| final String leftTableName = left.field0; | ||
| final String platformId = this.getPlatform().getPlatformId(); | ||
| final String leftTableName = JdbcParquetSource.resolveSourceName(configuration, platformId, left.field0); |
There was a problem hiding this comment.
Why is this specific to the parquet sources? what if the operator is used with table source? wouldn't this break?
There was a problem hiding this comment.
This was originally introduced as a minimal way to keep the relation names used by the join descriptors consistent with the configured Parquet source mappings. Regular table names currently fall back unchanged, but I agree that coupling the generic JDBC join operator to JdbcParquetSource is not the right abstraction. I’ll move the source-name resolution to the executor/source operators and keep the join operator source-agnostic.
| final String leftTableName = JdbcParquetSource.resolveSourceName(configuration, platformId, left.field0); | ||
| final String leftKeys = left.field1; | ||
| final String rightTableName = right.field0; | ||
| final String rightTableName = JdbcParquetSource.resolveSourceName(configuration, platformId, right.field0); |
There was a problem hiding this comment.
Hi Zoi, I moved the source-name resolution to the executor/source operators, so JdbcJoinOperator is now source-agnostic. I also added coverage to ensure regular table sources are not affected by Parquet mappings.
Summary
This PR adds support for executing
ParquetSourceon SQL-backed platforms by resolving Parquet inputs to platform-visible SQL relations.Supported platforms:
Changes
ParquetSourcesupport for SQL platforms.TrinoParquetSource,PrestoParquetSource, andBigQueryParquetSource.ParquetSource.