Repository navigation
Description
Description
Wayang supports several JDBC-backed platforms (PostgreSQL, SQLite3, generic JDBC, Trino, Presto, BigQuery) through wayang-jdbc-template, but has no ClickHouse platform. ClickHouse is a widely used column-oriented OLAP engine, and Wayang's SQL pushdown (filter, projection, join, aggregation, sort, table sink) suits it well.
I propose adding a wayang-platforms/wayang-clickhouse module that follows the structure of wayang-trino / wayang-presto / wayang-bigquery (all seven template operators, README, integration tests), rather than the smaller wayang-postgres module.
Proposed module layout
| Component | Class / file | Notes |
|---|---|---|
| Entry point | ClickHouse.java |
plugin(), conversionPlugin(), platform() |
| Platform | ClickHousePlatform extends JdbcPlatformTemplate |
name ClickHouse, config id clickhouse, official com.clickhouse:clickhouse-jdbc driver (Apache 2.0) |
| Operators | ClickHouse{TableSource,Filter,Projection,Join,GlobalReduce,ReduceBy,Sort,TableSink}Operator |
extend the existing Jdbc* base classes |
| Mappings | one per operator + Mappings.ALL |
same as Trino |
| Channels | ChannelConversions |
SQL -> Java stream, SQL -> Spark RDD |
| Plugins | ClickHousePlugin, ClickHouseConversionsPlugin |
used via .withPlugin(ClickHouse.plugin()) |
| Config | wayang-clickhouse-defaults.properties |
wayang.clickhouse.jdbc.{url,user,password}, cost parameters |
| Build | wayang-platforms/pom.xml |
new <module> |
ClickHouse-specific points
The generic SQL built in JdbcExecutor.createSqlString (SELECT ... FROM ... JOIN ... ON ... WHERE ... GROUP BY ... ORDER BY, no trailing ;) is valid ClickHouse SQL, so no changes to wayang-jdbc-template are planned. The dialect-specific work is:
- Table sink engine.
JdbcTableSinkOperatoremitsCREATE TABLE <t> AS <select>inoverwritemode. ClickHouse needs an engine for this, e.g.CREATE TABLE <t> ENGINE = MergeTree ORDER BY tuple() AS SELECT .... Since the engine clause must come beforeAS,ClickHouseTableSinkOperatorwould overridecreateSqlClause, with a configurable engine (e.g.wayang.clickhouse.tablesink.engine, defaultMergeTree ORDER BY tuple()).INSERT INTO <t> SELECT ...andDROP TABLE IF EXISTSwork unchanged. - Identifier quoting.
SqlIdentifierUtils.escapeAndQuoteIdentifierreads the quote character and case folding fromDatabaseMetaData. ClickHouse identifiers are case-sensitive; tests will coverdatabase.tablenames. - Type mapping.
SqlToStreamOperatorusesResultSet.getObject, so ClickHouse types such asUInt64(BigInteger),Date/DateTime,Nullable(T)andLowCardinality(String)appear as-is inRecords. I will document the mapping and test the common types. - Joins and NULLs. ClickHouse builds the hash table from the right side of a join, and columns are non-nullable by default (
join_use_nulls = 0). Behaviour will be documented; no changes to the join SQL are planned.
Testing plan
- Unit tests for the generated ClickHouse SQL (sink with engine clause, quoting) that need no server.
ClickHouseOperatorsITmodelled onTrinoOperatorsIT: covers all seven operators, creates and drops its own fixture tables, and is skipped (not failed) when no server is reachable (CLICKHOUSE_HOST/CLICKHOUSE_PORT).- No docker-compose or setup folders in the source tree, per Nonuniform conventions of providing benchmark/demo/example of running Wayang with different platforms #796.
Documentation and examples
wayang-platforms/wayang-clickhouse/README.mdClickHouseDemoinwayang-applicationspluswayang-applications/clickhouse.md- Add ClickHouse to the platform list in
wayang-docs/src/main/resources/index.md
Out of scope (possible follow-ups)
- Measured cost-model parameters (initial values will be uncalibrated, as in
wayang-postgres); aClickHouseCostPilotITfollowingguides/cost-profiling.mdcan follow. - Registering ClickHouse in
SqlContext,JsonPlanBuilder, orMultiContextDeserializer, which currently hard-code Postgres/SQLite. - Native (non-JDBC) protocol and ClickHouse-specific functions beyond pass-through SQL predicates.
Proposed PR sequence
To keep reviews small, I'd split this into sequential PRs. Each one builds and passes tests on its own.
| PR | Scope | Depends on |
|---|---|---|
| 1 | Module skeleton: pom.xml, ClickHousePlatform, ClickHouse entry point, ClickHouseExecutionOperator, ClickHouseTableSource, channel conversions, plugins, defaults properties, filter + projection mappings, SQL-generation unit tests, register in wayang-platforms/pom.xml |
- |
| 2 | Join, GlobalReduce, ReduceBy and Sort operators + mappings, ClickHouseOperatorsIT (skipped without a server) |
PR 1 |
| 3 | ClickHouseTableSinkOperator with configurable ENGINE clause, sink tests |
PR 1 |
| 4 | Module README, ClickHouseDemo + clickhouse.md in wayang-applications, docs index entry |
PRs 1-3 |
| 5 (optional) | Cost calibration via ClickHouseCostPilotIT and measured defaults |
PRs 1-3 |
PRs that depend on an earlier one will be opened as drafts marked "Depends on #X" and rebased after it merges.
Questions for maintainers
- Is the Trino/Presto/BigQuery module structure the preferred reference for a new SQL platform?
- Is a configurable sink engine acceptable, or should the sink only support
INSERT INTOinto pre-created tables? - Are integration tests that skip when no server is available acceptable for CI, or is a Testcontainers-based test preferred?
I'd like to work on this and will open a PR once the approach is agreed.
Activity
ekas-7 commentedon Nov 3, 2026
Hi @zkaoudi @juripetersen,
Following the contributing guidelines, I first introduced this proposal on the dev mailing list: <link to your thread on lists.apache.org/list.html?dev@wayang.apache.org>
Since this adds a new platform module, I've proposed splitting it into small sequential PRs (see "Proposed PR sequence" above) instead of one large PR. I'll wait for feedback on the approach, especially the open questions about module structure, the sink engine clause, and integration testing, before opening PR 1.
@juripetersen, following #796, I plan to put the ClickHouse demo in
wayang-applicationsand connect throughwayang.clickhouse.jdbc.*properties, without adding docker-compose or setup folders. Please let me know if these conventions have changed.Thanks!
zkaoudi commentedon Nov 4, 2026
+1 for supporting clickhouse
AliRana30 commentedon Nov 10, 2026
Hey @ekas-7 have you started working on the click house implementation? Can I collaborate with you in this process? If yes than we can discuss it in detail.
Kaileshwar16 commentedon Nov 10, 2026
I have a few thoughts on this. adding clickhouse or duckdb (#794) looks interesting, but since Wayang already has generic JDBC support, Im curious what a separate module would add. I think it could be better to have clear semantics before starting the implementation, otherwise the review could become quite difficult as the changes grow.
ekas-7 commentedon Nov 10, 2026
Hi @AliRana30, @Kaileshwar16,
Thanks for chiming in.
Regarding generic JDBC vs. a dedicated module (@Kaileshwar16):
The rationale matches why Wayang maintains dedicated modules for Trino, Presto, and BigQuery rather than relying on wayang-jdbc-template alone:
Engine semantics: ClickHouse DDL requires explicit storage engines (e.g., CREATE TABLE ... ENGINE = MergeTree ORDER BY tuple() AS SELECT ...). Generic JDBC does not support this syntax pattern for table sinks without engine-specific overrides.
Type system and dialect boundaries: Dedicated mapping ensures types like UInt64, LowCardinality, and Nullable translate correctly to Wayang streams/RDD channels, alongside ClickHouse-specific quoting rules and join nullability semantics.
Targeted cost modeling & pushdown optimization: A distinct platform identity is required to configure dialect-specific execution costs and accurately steer plan optimization toward column-oriented storage.
The implementation semantics are already isolated and scoped to avoid cross-module churn—it follows the established platform template without modifying wayang-jdbc-template.
Next steps (@AliRana30):
I already have the architectural plan and initial implementation underway locally following the phased PR sequence outlined above. To keep the review process tight and avoid coordination overhead, I will be driving the implementation of PR 1 through completion myself.
I'll push the draft for PR 1 shortly so everyone can review the concrete code against the proposal.
ekas-7 commentedon Nov 10, 2026
@zkaoudi Thanks for the +1! Could you please assign this issue to me? I'll start with PR 1 right away.
AliRana30 commentedon Nov 10, 2026
@ekas-7 Yeah I'm not gonna open any PR before what you plan and any approval from @zkaoudi . I will wait if you need help.
Complete ClickHouse platform: remaining operators, sink ENGINE, IT, a…
ekas-7 commentedon Nov 13, 2026
Hi @zkaoudi,
Thank you for the +1 on this. The implementation is now up in #877, and I would be grateful if you could take a look whenever you have time.
A short summary of what the PR contains:
wayang-platforms/wayang-clickhouse, following the Trino module layout: TableSource, Filter, Projection, Join, GlobalReduce, ReduceBy, Sort and TableSink.CREATE TABLE ... ENGINE = MergeTree ORDER BY tuple() AS SELECT ..., with the engine overridable per sink.ClickHouseOperatorsIT(11 tests), which I ran against ClickHouse 24.12 and which skips when no server is reachable.ClickHouseDemoandclickhouse.mdinwayang-applications, and the docs index entry. No docker-compose or setup folders, per Nonuniform conventions of providing benchmark/demo/example of running Wayang with different platforms #796.One thing I should flag: in this issue I proposed splitting the work into several small PRs, but #877 combines PRs 1–4. The pieces turned out to be hard to test in isolation (the integration test needs the sink, and the sink needs the other operators), so I kept them together. I realise that makes it a larger review. If you would prefer it split into smaller PRs, I am very happy to do that; just let me know.
The questions from the issue description are still open, and I will follow whatever you decide:
INSERT INTOpre-created tables?Cost calibration (
ClickHouseCostPilotIT) is not included and can follow separately, as noted under "out of scope".It also looks like the CI workflows on the PR are waiting for approval to run. Could you approve them when convenient?
Thank you for your time, and for any feedback.