Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 34 additions & 15 deletions INTEGRATION-TESTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,16 @@
limitations under the License.
-->

# Integration Tests – Presto
# Integration Tests – SQL Platforms

End-to-end tests that drive the Presto platform operators (TableSource, Filter,
Projection, Join) through the Wayang API against a live PrestoDB cluster.
End-to-end tests that drive JDBC platform operators through the Wayang API
against a user-managed database. Each suite **skips** (it does not fail) when
the server is unreachable.

| Test | Module | Needs |
|------|--------|-------|
| `AllOperatorsIT` | `wayang-presto` | a local Presto (Docker) |

The test **skips** (it does not fail) when Presto is unreachable.
| `PrestoOperatorsIT` | `wayang-presto` | a Presto deployment |
| `ClickHouseOperatorsIT` | `wayang-clickhouse` | a ClickHouse deployment |

---

Expand Down Expand Up @@ -74,17 +74,36 @@ Connection settings are supplied through `PRESTO_HOST`, `PRESTO_PORT`, and

---

## ClickHouse

The test creates and seeds its own `wayang_it` database/tables in a user-managed
ClickHouse deployment, then drops the fixtures afterward.

```bash
CLICKHOUSE_HOST=localhost CLICKHOUSE_PORT=8123 CLICKHOUSE_USER=default \
JAVA_HOME=/path/to/jdk-17 \
./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-clickhouse -am \
-Dtest=ClickHouseOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \
-DfailIfNoTests=false \
-Drat.skip=true -Dlicense.skip=true -Dmaven.javadoc.skip=true test
```

Connection settings are supplied through `CLICKHOUSE_HOST`, `CLICKHOUSE_PORT`,
`CLICKHOUSE_USER`, and optional `CLICKHOUSE_PASSWORD`. Applications use
`wayang.clickhouse.jdbc.*` properties instead.

## Notes

- **Pushdown is cost-gated.** On tiny tables Wayang's optimizer prefers a full
scan + Java-side filter/projection, so pushdown only appears once a table is
large enough (hence the test scales to 120k rows). Each test asserts both correct
results and that the expected SQL reached Presto (`system.runtime.queries`).
large enough (hence the test scales to 120k rows). Presto tests also assert
that the expected SQL reached the server (`system.runtime.queries`).
- **Join.** A JDBC join is verified through the operator's SQL-clause contract
executed on Presto, not the high-level `WayangContext` API — the logical
`JoinOperator` emits `Tuple2<Record,Record>`, which cannot connect to a `Record`
sink before the SQL pushdown flattens it.
- **Trailing semicolons.** Presto's SQL parser rejects a trailing `;` in
`executeQuery`, so this branch also carries the jdbc-template change that stops
emitting one (shared with the other JDBC platforms; Postgres/SQLite tolerate its
absence).
executed on the platform, not the high-level `WayangContext` API alone — the
logical `JoinOperator` emits `Tuple2<Record,Record>`, which cannot connect to a
`Record` sink before the SQL pushdown flattens it.
- **Trailing semicolons.** Strict SQL parsers (Presto, Trino, BigQuery,
ClickHouse JDBC in some modes) reject a trailing `;` in `executeQuery`, so
`wayang-jdbc-template` does not emit one.
- **ClickHouse table sinks.** Overwrite mode emits
`CREATE TABLE ... ENGINE = MergeTree ORDER BY tuple() AS SELECT ...`.
1 change: 1 addition & 0 deletions wayang-applications/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ particular application are listed here.
## SQL Platform Examples

- [BigQuery](bigquery.md)
- [ClickHouse](clickhouse.md)
- [DuckDB](duckdb.md)
- [Presto](presto.md)
- [Trino](trino.md)
97 changes: 97 additions & 0 deletions wayang-applications/clickhouse.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

# ClickHouse Example

This example runs filter and projection plans against an existing ClickHouse
table. Wayang pushes `region = 'AMER'` and the selected columns into SQL
executed by ClickHouse.

## Prerequisites

- Java 17
- A ClickHouse deployment reachable from the machine running Wayang
- A ClickHouse user with `SELECT` permission on the example table

Wayang does not start ClickHouse.

## Prepare the table

Using the ClickHouse client, replace `default` with a database available in
your deployment, then run:

```sql
CREATE DATABASE IF NOT EXISTS default;

CREATE TABLE IF NOT EXISTS default.orders (
order_id Int64,
region String,
product String,
amount Float64,
order_date String
) ENGINE = MergeTree ORDER BY order_id;

INSERT INTO default.orders VALUES
(1, 'AMER', 'book', 25.50, '2026-01-10'),
(2, 'EMEA', 'desk', 300.00, '2026-01-11'),
(3, 'AMER', 'chair', 85.25, '2026-01-12');
```

## Configure Wayang

Create a properties file such as `/tmp/clickhouse-example.properties`:

```properties
wayang.clickhouse.jdbc.url = jdbc:clickhouse://clickhouse.example.com:8123/default
wayang.clickhouse.jdbc.user = default
wayang.clickhouse.jdbc.password =
wayang.clickhouse.demo.table = default.orders
```

Change the endpoint, credentials, database, and table for your deployment.

To override the table-sink ENGINE for `writeTable(..., "overwrite", ...)`, put
`wayang.clickhouse.tablesink.engine` on the `Properties` argument (default is
`MergeTree ORDER BY tuple()`).

## Build and run

Build the application and its dependencies from the repository root:

```bash
./mvnw -Pskip-prerequisite-check -pl wayang-applications -am \
-DskipTests -Dpython.worker.tests.skip=true install
```

Run the example:

```bash
./mvnw -Pskip-prerequisite-check -pl wayang-applications exec:java \
-Dexec.mainClass=org.apache.wayang.applications.ClickHouseDemo \
"-Dexec.args=file:///tmp/clickhouse-example.properties"
```

Both sections should return the two `AMER` rows. The projection section returns
only `region`, `product`, and `amount`.

## Troubleshooting

- **Connection refused:** verify the host, HTTP port (default `8123`), and
network access from Wayang.
- **Authentication failure:** check the user and password, or JDBC URL options
required by the deployment.
- **Table not found:** use a fully qualified `database.table` name.
5 changes: 5 additions & 0 deletions wayang-applications/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,11 @@
<artifactId>wayang-trino</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.wayang</groupId>
<artifactId>wayang-clickhouse</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.wayang</groupId>
<artifactId>wayang-core</artifactId>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.wayang.applications;

import org.apache.wayang.basic.data.Record;
import org.apache.wayang.basic.function.ProjectionDescriptor;
import org.apache.wayang.basic.operators.FilterOperator;
import org.apache.wayang.basic.operators.LocalCallbackSink;
import org.apache.wayang.basic.operators.MapOperator;
import org.apache.wayang.basic.types.RecordType;
import org.apache.wayang.clickhouse.ClickHouse;
import org.apache.wayang.clickhouse.operators.ClickHouseTableSource;
import org.apache.wayang.core.api.Configuration;
import org.apache.wayang.core.api.WayangContext;
import org.apache.wayang.core.function.PredicateDescriptor;
import org.apache.wayang.core.plan.wayangplan.WayangPlan;
import org.apache.wayang.core.types.DataSetType;
import org.apache.wayang.java.Java;

import java.util.ArrayList;
import java.util.List;

/**
* Configurable example for the Wayang ClickHouse connector.
*
* <p>See {@code wayang-applications/clickhouse.md} for configuration and usage.
*/
public class ClickHouseDemo {

private static Configuration configuration;
private static String sourceTable;

public static void main(String[] args) throws Exception {
if (args.length != 1) {
throw new IllegalArgumentException("Usage: ClickHouseDemo <configuration URL>");
}
configuration = new Configuration(args[0]);
configuration.getStringProperty("wayang.clickhouse.jdbc.url");
sourceTable = configuration.getStringProperty(
"wayang.clickhouse.demo.table", "default.orders"
);
filterDemo();
projectionDemo();
}

static void filterDemo() throws Exception {
System.out.println("ClickHouse filter pushdown");
System.out.println();
System.out.println(" Operator: FilterOperator -> ClickHouseFilterOperator");
System.out.println(" SQL sent: SELECT * FROM " + sourceTable);
System.out.println(" WHERE region = 'AMER'");
System.out.println();

WayangContext wayang = buildWayang();

List<Record> results = new ArrayList<>();
ClickHouseTableSource source = new ClickHouseTableSource(
sourceTable, "order_id", "region", "product", "amount", "order_date"
);
FilterOperator<Record> filter = new FilterOperator<>(
new PredicateDescriptor<>(
r -> "AMER".equals(r.getField(1)), Record.class
).withSqlImplementation("region = 'AMER'")
);
LocalCallbackSink<Record> sink = LocalCallbackSink.createCollectingSink(results, Record.class);
source.connectTo(0, filter, 0);
filter.connectTo(0, sink, 0);

wayang.execute("ClickHouse-Filter-Demo", new WayangPlan(sink));

System.out.println(" Results returned by Wayang:");
System.out.printf(" %-10s %-6s %-10s %10s %-12s%n",
"order_id", "region", "product", "amount", "order_date");
System.out.println(" " + repeat('-', 54));
for (Record r : results) {
System.out.printf(" %-10s %-6s %-10s %10s %-12s%n",
r.getField(0), r.getField(1), r.getField(2), r.getField(3), r.getField(4));
}
System.out.println();
System.out.printf(" %d AMER rows returned.%n", results.size());
System.out.println();
}

static void projectionDemo() throws Exception {
System.out.println("ClickHouse projection pushdown");
System.out.println();
System.out.println(" Operators: FilterOperator -> ClickHouseFilterOperator");
System.out.println(" MapOperator -> ClickHouseProjectionOperator");
System.out.println(" SQL sent: SELECT region, product, amount");
System.out.println(" FROM " + sourceTable);
System.out.println(" WHERE region = 'AMER'");
System.out.println();

WayangContext wayang = buildWayang();

List<Record> results = new ArrayList<>();
ClickHouseTableSource source = new ClickHouseTableSource(
sourceTable, "order_id", "region", "product", "amount", "order_date"
);
FilterOperator<Record> filter = new FilterOperator<>(
new PredicateDescriptor<>(
r -> "AMER".equals(r.getField(1)), Record.class
).withSqlImplementation("region = 'AMER'")
);
MapOperator<Record, Record> projection = new MapOperator<>(
ProjectionDescriptor.createForRecords(
new RecordType("order_id", "region", "product", "amount", "order_date"),
"region", "product", "amount"),
DataSetType.createDefault(Record.class),
DataSetType.createDefault(Record.class)
);
LocalCallbackSink<Record> sink = LocalCallbackSink.createCollectingSink(results, Record.class);
source.connectTo(0, filter, 0);
filter.connectTo(0, projection, 0);
projection.connectTo(0, sink, 0);

wayang.execute("ClickHouse-Projection-Demo", new WayangPlan(sink));

System.out.println(" Results returned by Wayang (projected columns only):");
System.out.printf(" %-6s %-10s %10s%n", "region", "product", "amount");
System.out.println(" " + repeat('-', 30));
for (Record r : results) {
System.out.printf(" %-6s %-10s %10s%n", r.getField(0), r.getField(1), r.getField(2));
}
System.out.println();
System.out.printf(" %d AMER rows returned with 3 columns.%n", results.size());
System.out.println();
}

private static WayangContext buildWayang() {
return new WayangContext(configuration)
.withPlugin(Java.basicPlugin())
.withPlugin(ClickHouse.plugin());
}

private static String repeat(char c, int n) {
StringBuilder sb = new StringBuilder(n);
for (int i = 0; i < n; i++) {
sb.append(c);
}
return sb.toString();
}
}
2 changes: 2 additions & 0 deletions wayang-docs/src/main/resources/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ Wayang has built in support for the following frameworks:
- Trino
- Presto
- Google BigQuery
- ClickHouse
- TensorFlow (Deep Learning)
- Apache Giraph v1.2.0

Expand Down Expand Up @@ -95,6 +96,7 @@ Add the platform adapters you wish to target:
- `wayang-postgres` / `wayang-sqlite3` / `wayang-generic-jdbc`: Relational database platforms
- `wayang-trino` / `wayang-presto`: Distributed SQL query engines
- `wayang-bigquery`: Google BigQuery cloud data warehouse
- `wayang-clickhouse`: ClickHouse OLAP database
- `wayang-tensorflow`: TensorFlow deep learning platform
- `wayang-giraph`: Graph processing platform

Expand Down
1 change: 1 addition & 0 deletions wayang-platforms/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
<module>wayang-presto</module>
<module>wayang-trino</module>
<module>wayang-duckdb</module>
<module>wayang-clickhouse</module>
<module>wayang-tensorflow</module>
<module>wayang-semantic</module>
</modules>
Expand Down
Loading