Skip to content

Add wayang-clickhouse platform support (#871) - #877

Open
ekas-7 wants to merge 7 commits into
apache:mainfrom
ekas-7:feature/clickhouse-platform-pr1
Open

ekas-7 wants to merge 7 commits into
apache:mainfrom
ekas-7:feature/clickhouse-platform-pr1

Conversation

@ekas-7

@ekas-7 ekas-7 commented Nov 10, 2026 •

Copy link
Copy Markdown

Summary

  • Add wayang-platforms/wayang-clickhouse as a JDBC platform module (Trino-style layout).
  • Cover TableSource, Filter, Projection, Join, GlobalReduce, ReduceBy, Sort, and TableSink.
  • TableSink overwrite mode emits CREATE TABLE ... ENGINE = MergeTree ORDER BY tuple() AS SELECT .... The engine can be overridden per sink by setting wayang.clickhouse.tablesink.engine on the TableSink / writeTable Properties.
  • Uses com.clickhouse:clickhouse-jdbc:0.11.1 (all classifier, transitive dependencies excluded). This release relocates its bundled ANTLR, so it does not clash with the ANTLR 4.9.3 runtime used by Wayang core.
  • Unit tests for SQL generation / plugin registration; ClickHouseOperatorsIT skips when no server is reachable.
  • Module README, ClickHouseDemo + clickhouse.md in wayang-applications, and docs index entry.
  • Merged with main (keeps both wayang-duckdb and wayang-clickhouse).

Closes #871.

Test plan

  • ./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-clickhouse -am verify on JDK 17: build, RAT licence check and 41 unit tests pass.
  • ClickHouseOperatorsIT against a live ClickHouse 24.12.6 (CLICKHOUSE_HOST / CLICKHOUSE_PORT, passwordless default user): 11/11 pass, covering all eight operators, two JavaPlanBuilder plans and the ClickHouse-to-Java type mapping (UInt64, Date, DateTime, Nullable, LowCardinality).
  • ClickHouseDemo run as documented in wayang-applications/clickhouse.md.
  • Reviewer: confirm sink ENGINE clause and no docker-compose / setup folders (per Nonuniform conventions of providing benchmark/demo/example of running Wayang with different platforms #796)

Test results

Suite Tests Result
ClickHousePlatformTest 2 pass
ClickHousePluginTest 2 pass
ClickHouseSqlGenerationTest 10 pass
ClickHouseTableSinkCopyTest 1 pass
ClickHouseMappingsTest 14 pass
ClickHouseOperatorsTest 12 pass
ClickHouseOperatorsIT (live ClickHouse 24.12.6) 11 pass

SQL executed by the integration test, as logged by JdbcExecutor:

CREATE TABLE `wayang_it`.`operator_result` ENGINE = MergeTree ORDER BY tuple() AS SELECT region, amount FROM wayang_it.orders WHERE region = 'AMER'
CREATE TABLE `wayang_it`.`operator_result` ENGINE = MergeTree ORDER BY tuple() AS SELECT order_id, customer_id, region, amount, cust_id, name, tier FROM wayang_it.orders JOIN wayang_it.customers ON wayang_it.customers.cust_id=wayang_it.orders.customer_id
CREATE TABLE `wayang_it`.`operator_result` ENGINE = MergeTree ORDER BY tuple() AS SELECT region,SUM(amount) AS total_amount FROM wayang_it.orders GROUP BY region
CREATE TABLE `wayang_it`.`operator_result` ENGINE = MergeTree ORDER BY tuple() AS SELECT * FROM wayang_it.orders ORDER BY amount ASC

Coverage

JaCoCo 0.8.16 over org.apache.wayang.clickhouse.* (22 classes):

Unit tests only (41) Unit + ClickHouseOperatorsIT (52)
Lines 175/178 (98.3%) 175/178 (98.3%)
Branches 28/32 (87.5%) 28/32 (87.5%)
Methods 98/101 (97.0%) 98/101 (97.0%)
Classes 22/22 (100%) 22/22 (100%)

The three uncovered lines are the implicit constructors of the static holder classes ClickHouse, Mappings and ChannelConversions.

Notes:

  • The root pom's surefire configuration only includes *Test / *Tests, so ClickHouseOperatorsIT is not run by mvn verify in CI (same as TrinoOperatorsIT / PrestoOperatorsIT); run it with -Dtest=ClickHouseOperatorsIT.
  • The JaCoCo agent was attached via JAVA_TOOL_OPTIONS, because the surefire argLine in the root pom replaces the one set by jacoco:prepare-agent.

…pport.

Introduces the ClickHouse JDBC module following the Trino/Postgres plugin pattern so filter/projection SQL can be pushed down, as the first slice of apache#871.
Resolve the wayang-platforms/pom.xml conflict by keeping both wayang-duckdb and wayang-clickhouse modules.
…nd docs.

Adds join/reduce/sort/table-sink mappings, ClickHouseOperatorsIT, README, and a wayang-applications demo so apache#871 is covered in this PR.
@ekas-7 ekas-7 changed the title Add wayang-clickhouse platform skeleton (PR 1 of #871) Add wayang-clickhouse platform support (#871) Nov 13, 2026
Preserve custom ENGINE across createCopy, clarify that the engine key is a TableSink property, document ClickHouseOperatorsIT, and scale fixtures with CROSS JOIN numbers().
Bump clickhouse-jdbc to 0.11.1, whose shaded jar relocates ANTLR, and exclude
its transitive dependencies. The 0.9.0 jar shipped an unrelocated ANTLR 4.13
that shadowed Wayang's 4.9.3 runtime and broke the mathex parser.

Always pass the password property (the driver requires it even when empty) and
alias the numbers() table function in the fixture SQL, so the IT actually runs
against a passwordless ClickHouse instead of silently skipping.
Probe the connection with SELECT 1 first and skip the class only if that
fails. Fixture statements now run outside the catch, so a broken fixture
fails the run instead of being reported as a skip.
ClickHouseMappingsTest applies every mapping to a logical plan and checks the
replacement operator, including the cases that must not be mapped.
ClickHouseOperatorsTest covers copies, channels, the sink ENGINE fallbacks and
that the defaults file provides a cost model for every operator.
ClickHouseOperatorsIT now reads UInt64, Date, DateTime, Nullable and
LowCardinality columns into Records; the README documents the Java types.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add ClickHouse platform support (wayang-clickhouse)

1 participant