Skip to content

Add SQL platform support for ParquetSource - #793

Open
JunWang222 wants to merge 9 commits into
apache:mainfrom
JunWang222:feature/parquet-sql-independent-platforms-clean
Open

JunWang222 wants to merge 9 commits into
apache:mainfrom
JunWang222:feature/parquet-sql-independent-platforms-clean

Conversation

@JunWang222

Copy link
Copy Markdown
Contributor

Summary

This PR adds support for executing ParquetSource on SQL-backed platforms by resolving Parquet inputs to platform-visible SQL relations.

Supported platforms:

  • Trino
  • Presto
  • BigQuery

Changes

  • Add JDBC-based ParquetSource support for SQL platforms.
  • Add TrinoParquetSource, PrestoParquetSource, and BigQueryParquetSource.
  • Add platform mappings for ParquetSource.
  • Add Parquet source configuration keys for platform-specific relation mapping and optional relation creation.
  • Add BigQuery platform support needed for SQL operator execution.
  • Add integration coverage for Parquet joins and platform alternatives.

JunWang222 and others added 5 commits August 12, 2026 17:08
…-independent-platforms-clean

# Conflicts:
#	wayang-platforms/wayang-jdbc-template/src/main/java/org/apache/wayang/jdbc/execution/JdbcExecutor.java
#	wayang-platforms/wayang-trino/src/main/resources/wayang-trino-defaults.properties
…m/apache/wayang into feature/parquet-sql-independent-platforms-clean

# Conflicts:
#	wayang-platforms/wayang-bigquery/src/main/resources/wayang-bigquery-defaults.properties
#	wayang-platforms/wayang-jdbc-template/src/main/java/org/apache/wayang/jdbc/execution/JdbcExecutor.java
#	wayang-platforms/wayang-presto/src/main/resources/wayang-presto-defaults.properties
final Tuple<String, String> right = this.keyDescriptor1.getSqlImplementation();
final String leftTableName = left.field0;
final String platformId = this.getPlatform().getPlatformId();
final String leftTableName = JdbcParquetSource.resolveSourceName(configuration, platformId, left.field0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this specific to the parquet sources? what if the operator is used with table source? wouldn't this break?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was originally introduced as a minimal way to keep the relation names used by the join descriptors consistent with the configured Parquet source mappings. Regular table names currently fall back unchanged, but I agree that coupling the generic JDBC join operator to JdbcParquetSource is not the right abstraction. I’ll move the source-name resolution to the executor/source operators and keep the join operator source-agnostic.

final String leftTableName = JdbcParquetSource.resolveSourceName(configuration, platformId, left.field0);
final String leftKeys = left.field1;
final String rightTableName = right.field0;
final String rightTableName = JdbcParquetSource.resolveSourceName(configuration, platformId, right.field0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Zoi, I moved the source-name resolution to the executor/source operators, so JdbcJoinOperator is now source-agnostic. I also added coverage to ensure regular table sources are not affected by Parquet mappings.

@JunWang222
JunWang222 requested a review from zkaoudi November 7, 2026 04:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants