Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 115 additions & 0 deletions python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,83 @@ configuration.set_property("wayang.api.python.port", "8080")
The same values can be supplied through environment variables:
`WAYANG_API_URL`, or `WAYANG_API_HOST` and `WAYANG_API_PORT`.

## In-memory collections

`load_collection()` snapshots a Python list or tuple and returns a `DataQuanta`:

```python
from pywy.dataquanta import WayangContext
from pywy.platforms.java import JavaPlugin

ctx = WayangContext().register({JavaPlugin})
ctx.load_collection([1, 2, 3]).store_textfile("file:///tmp/numbers.txt")
```

Supported values are built-in strings, booleans, signed 64-bit integers, finite
floats, and nested lists or tuples. Tuples become lists. Empty and mixed
collections are supported. Input order and nested structure are preserved by
the source; subsequent operators/platforms determine result ordering. `None`,
mappings, sets, generators, custom objects, cyclic containers, out-of-range
integers, and non-finite floats are rejected before submission.

The current Python worker passes records to UDFs as strings, including records
from collection sources. Convert numeric inputs explicitly:

```python
(ctx.load_collection([1, 2, 3])
.filter(lambda value: int(value) > 1, str)
.map(lambda value: int(value) * 2, str, int)
.store_textfile("file:///tmp/doubled.txt"))
```

This example uses the existing fusion of adjacent Python transformations into
one worker call. The standalone Python filter and primitive-typed map-to-text
paths have existing limitations (empty filter output and incompatible sink
types, respectively), also reproducible with text-file sources. Collection
input does not change those paths.

Nested values remain structured on the JVM, but reach Python UDFs and text sinks
in Java list notation, such as `[1, 2]`. This is not a JSON round-trip format.
Text sinks use the existing JVM string representation, one record per line;
embedded newlines in strings produce additional lines. File paths refer to the
backend's filesystem. Collection data is held in memory and sent in one JSON
request, so this source is intended for small local datasets.

This API requires a backend supporting literal `inputCollection.data.values`.
The backend continues to accept existing `inputCollection.data.udf` plans.

### Collecting results

Use `collect()` to execute a pipeline and receive a Python list:

```python
result = (ctx.load_collection([1, 2, 3])
.map(lambda value: int(value) * 2, str, int)
.collect())
# [2, 4, 6]
```

Empty results return `[]`. Supported result values are the same as collection
inputs: strings, booleans, signed 64-bit integers, finite floats, and nested
lists/tuples (tuples become lists). Unsupported values cause an error instead
of being converted to strings. Each call executes the pipeline again without
changing the graph. Result order depends on the operators and platform.

Adjacent Python transformations at the end of a collected pipeline run together
and encode their results before crossing the worker boundary. This preserves
their runtime types, including nested values, and supports standalone maps and
filters. Inputs to those transformations still follow the existing string-based
worker behavior: filtering `[1, 2, 3]` with `lambda x: int(x) > 1` yields
`["2", "3"]`. Map with `int` before filtering to retain numeric results in the
fused pipeline. Earlier worker boundaries, for example before a join, retain
their existing limitations; collecting cannot restore types already lost there.

The backend, Python client, and worker must all include collect support.
Execution failures raise `requests.HTTPError` with the backend error response;
invalid or unsupported responses raise `ValueError`. Results are materialized
in backend and client memory, so use this method for results that fit in memory.
`store_textfile()` and legacy REST responses retain their existing behavior.

## Docker

The Python API can be run with the Wayang REST API as a separate container.
Expand Down Expand Up @@ -148,3 +225,41 @@ docker network rm wayang-python

### Testing python code
You can run the python tests by using pytest, the requirements for the tests are listed in `python/src/pywy/requirements.txt`. To run the tests navigate to the base wayang folder, e.g. `/var/www/html` and run `pytest -s python/src/pywy` if you need to pass a specific configuration for your use case you can also add a config flag `pytest -s --config=pathToYourConfig python/src/pywy/`

The collection unit tests run without a backend:

```bash
PYTHONPATH=python/src pytest -q python/src/pywy/tests/test_load_collection.py \
python/src/pywy/tests/test_collect.py
```

For the collection integration tests, build and install the JSON API and its
dependencies with JDK 17 (`./mvnw install -pl :wayang-api-json -am -DskipTests`).
Start `org.apache.wayang.api.json.Main` as described in the JSON API README, or use:

```bash
./mvnw -pl :wayang-api-json exec:java \
-Dexec.mainClass=org.apache.wayang.api.json.Main -Dexec.args=8080 \
-Dwayang.configuration=file:///absolute/path/to/wayang.properties
```

The properties file must set `wayang.api.python.path` to the Python executable,
`wayang.api.python.worker` to the absolute path of
`python/src/pywy/execution/worker.py`, and `wayang.api.python.env.path` to the
absolute path of `python/src`. Run from the repository root against a backend
on the same machine (the tests read its output files):

```bash
PYTHONPATH=python/src \
PYWAYANG_INTEGRATION_URL=http://localhost:8080/wayang-api-json/submit-plan/json \
pytest -q python/src/pywy/tests/test_load_collection_integration.py \
python/src/pywy/tests/test_collect_integration.py
```

Without `PYWAYANG_INTEGRATION_URL`, these integration tests explicitly skip.
The collect tests cover typed results, repeated execution, backend failures,
and compatibility with text sinks and legacy REST clients. Set
`PYWAYANG_TEST_SPARK=1` to also exercise collection on local Spark; the backend
must be configured to run Spark with the JVM options required by JDK 17.
The [wayang-submit launcher](../bin/wayang-submit) supplies these options. When
starting the backend with Maven, pass its `--add-opens` options in `MAVEN_OPTS`.
35 changes: 29 additions & 6 deletions python/src/pywy/core/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
from pywy.core.serializer import JSONSerializer
from pywy.graph.graph import WayangGraph
from pywy.graph.types import WGraphOfVec
from pywy.operators import SinkOperator, UnaryToUnaryOperator
from pywy.operators import PywyOperator, CollectionSource, UnaryToUnaryOperator


class Plugin:
Expand Down Expand Up @@ -64,13 +64,13 @@ class PywyPlan:
plugins : :obj:`set` of :py:class:`pywy.core.plugin.Plugin`
plugins is the set of possible platforms that can be uses to execute
the PywyPlan
sinks : :py:class:`typing.Iterable` of :py:class:`pywy.operators.sink.SinkOperator`
The list of sink operators, this describe the end of the pipeline, and
they are used to build the `graph`
sinks : :py:class:`typing.Iterable` of :py:class:`pywy.operators.base.PywyOperator`
Terminal operators used to build the graph. For collect(), the terminal
operator need not be a sink.
"""
graph: WayangGraph

def __init__(self, plugins: Set[Plugin], configuration: Configuration, sinks: Iterable[SinkOperator]):
def __init__(self, plugins: Set[Plugin], configuration: Configuration, sinks: Iterable[PywyOperator]):
"""basic Constructor of PywyPlan

this constructor set the plugins and sinks element, and it prepares
Expand All @@ -93,7 +93,7 @@ def set_graph(self):
"""
self.graph = WGraphOfVec(self.sinks)

def execute(self):
def execute(self, collect: bool = False):
""" Transform the plan into topologies to group pipelines into one
MapPartition operator
Transform the plan into json objects to send it to Wayang
Expand All @@ -109,6 +109,8 @@ def execute(self):

json_data["context"] = context
json_data["operators"] = []
if collect:
json_data["collect"] = True

nodes = []
pipeline = []
Expand All @@ -127,6 +129,9 @@ def execute(self):
json_data["operators"].append(serializer.serialize(operator))
pipeline = []

if pipeline:
json_data["operators"].append(serializer.serialize_pipeline(pipeline, json_output=collect))

api_url = self.configuration.get_property("wayang.api.python.url") or os.environ.get("WAYANG_API_URL")
if api_url is None:
host = (
Expand All @@ -145,3 +150,21 @@ def execute(self):
json_body = json.dumps(json_data)
print(json_body)
response = requests.post(api_url, headers=headers, json=json_data)
if collect:
if not response.ok:
raise requests.HTTPError(
f"Wayang execution failed (HTTP {response.status_code}): {response.text}",
response=response
)
try:
body = response.json()
except ValueError as error:
raise ValueError(
"Wayang collect expected a JSON response; check the backend version"
) from error
if not isinstance(body, dict) or not isinstance(body.get("results"), list):
raise ValueError("Wayang collect expected a JSON object with a results array")
try:
return CollectionSource._snapshot(body["results"], set(), "results")
except (TypeError, ValueError) as error:
raise ValueError(f"Unsupported Wayang collect response: {error}") from error
20 changes: 20 additions & 0 deletions python/src/pywy/core/functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
import json

from pywy.execution.util import JsonRecord
from pywy.operators.source import CollectionSource


class ChainedFunctions:
def __init__(self):
self.functions = []
Expand All @@ -27,3 +33,17 @@ def execute(self, input_data):
for function in self.functions:
result = function(result)
return result

def execute_json(self, input_data):
# Encode before the worker's legacy str()/NumPy conversion loses types.
try:
for value in self.execute(input_data):
value = CollectionSource._snapshot(value, set(), "collect result")
yield JsonRecord(
json.dumps(value, ensure_ascii=False, allow_nan=False).encode("utf-8")
)
except Exception as error:
# Preserve the Python cause instead of exposing only a socket EOF.
yield JsonRecord(
json.dumps({"error": f"{type(error).__name__}: {error}"}).encode("utf-8")
)
35 changes: 25 additions & 10 deletions python/src/pywy/core/serializer.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# limitations under the License.
#

from typing import Set, Iterable
from typing import Set, Dict
import json
import base64
import cloudpickle
Expand All @@ -28,14 +28,19 @@
from pywy.graph.graph import WayangGraph
from pywy.graph.types import WGraphOfVec, NodeOperator, NodeVec
from pywy.types import get_java_type, NDimArray, ndim_from_type
from pywy.operators import SinkOperator, UnaryToUnaryOperator, SourceUnaryOperator
from pywy.operators import PywyOperator, SinkOperator, UnaryToUnaryOperator, SourceUnaryOperator


class JSONSerializer:
id_table: Iterable[int]
id_table: Dict[PywyOperator, int]

def __init__(self, id_table: Iterable[int]):
def __init__(self, id_table: Dict[PywyOperator, int]):
self.id_table = id_table
# Forward links retain only the latest consumer, so derive edges from this plan's inputs.
self.consumers = {operator: [] for operator in id_table}
for operator in id_table:
for upstream in operator.inputOperator:
self.consumers[upstream].append(operator)

def serialize(self, operator):
json_operator = {}
Expand All @@ -48,7 +53,9 @@ def serialize(self, operator):
json_operator["input"] = []

if operator.cat != "output":
json_operator["output"] = list(map(lambda x: self.id_table[x], operator.outputOperator))
json_operator["output"] = [
self.id_table[output] for output in self.consumers[operator]
]
else:
json_operator["output"] = []

Expand All @@ -60,7 +67,10 @@ def serialize(self, operator):
if hasattr(operator, "output_type") and operator.output_type is not None:
json_operator["data"]["outputType"] = ndim_from_type(operator.output_type).to_json()

if operator.json_name == "filter":
if operator.json_name == "inputCollection":
json_operator["data"]["values"] = operator.values

elif operator.json_name == "filter":
json_operator["data"]["udf"] = base64.b64encode(cloudpickle.dumps(operator.use_predicate)).decode('utf-8')

elif operator.json_name == "reduceBy":
Expand Down Expand Up @@ -93,7 +103,7 @@ def serialize(self, operator):

return json_operator

def serialize_pipeline(self, pipeline):
def serialize_pipeline(self, pipeline, json_output=False):
assert len(pipeline) > 0

json_operator = {}
Expand All @@ -102,9 +112,11 @@ def serialize_pipeline(self, pipeline):
json_operator["cat"] = "unary"
json_operator["data"] = {}
json_operator["input"] = list(map(lambda x: self.id_table[x], pipeline[0].inputOperator))
json_operator["output"] = list(map(lambda x: self.id_table[x], pipeline[-1].outputOperator))
json_operator["output"] = [
self.id_table[output] for output in self.consumers[pipeline[-1]]
]

if len(pipeline) == 1:
if len(pipeline) == 1 and not json_output:
return self.serialize(pipeline[0])

chainedUdfs = ChainedFunctions()
Expand All @@ -114,6 +126,9 @@ def serialize_pipeline(self, pipeline):
if hasattr(operator, "get_udf"):
chainedUdfs.add_function(operator.get_udf)

json_operator["data"]["udf"] = base64.b64encode(cloudpickle.dumps(chainedUdfs.execute)).decode('utf-8')
udf = chainedUdfs.execute_json if json_output else chainedUdfs.execute
json_operator["data"]["udf"] = base64.b64encode(cloudpickle.dumps(udf)).decode('utf-8')
if json_output:
json_operator["data"]["jsonOutput"] = True

return json_operator
29 changes: 28 additions & 1 deletion python/src/pywy/dataquanta.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# limitations under the License.
#

from typing import Set, List, Optional, cast
from typing import Set, List, Optional, Tuple, Union, cast

from pywy.core.core import Plugin, PywyPlan
from pywy.operators.base import PO_T
Expand Down Expand Up @@ -56,6 +56,21 @@ def unregister(self, *plugins: Plugin):
def textfile(self, file_path: str) -> "DataQuanta[str]":
return DataQuanta(self, TextFileSource(file_path))

def load_collection(self, collection: Union[List[T], Tuple[T, ...]]) -> "DataQuanta[T]":
"""Create a source from a snapshot of an ordered list or tuple.

Values may be strings, booleans, signed 64-bit integers, finite floats,
or nested lists/tuples (tuples become lists). Empty collections are valid.
Other values, including None, mappings, sets, and cyclic containers,
raise TypeError or ValueError before a plan is submitted.

As with other sources, the current Python worker passes records to UDFs
as strings. Numeric UDFs must convert their input explicitly with int()
or float(). See the README for a fused transformation example and
existing worker limitations.
"""
return DataQuanta(self, CollectionSource(collection))

def parquet(
self, file_path: str, projection: Optional[List[str]] = None, column_names: Optional[List[str]] = None
) -> "DataQuanta[Record]":
Expand Down Expand Up @@ -219,6 +234,18 @@ def train_linear_svc(



def collect(self: "DataQuanta[T]") -> List[T]:
"""Execute this pipeline and return its results in memory.

Supports the same values as load_collection(); tuples become lists.
Result order depends on the operators and execution platform. Each call
executes the pipeline again. Backend failures raise requests.HTTPError;
invalid or unsupported results raise ValueError.
"""
return PywyPlan(
self.context.plugins, self.context.configuration, [self.operator]
).execute(collect=True)

def store_textfile(self: "DataQuanta[In]", path: str, input_type: GenericTco = None) -> None:
last: List[SinkOperator] = [
cast(
Expand Down
4 changes: 4 additions & 0 deletions python/src/pywy/execution/util.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@
#


class JsonRecord(bytes):
"""An encoded collect record, distinct from a legacy UDF returning bytes."""


class SpecialLengths(object):
END_OF_DATA_SECTION = -1
PYTHON_EXCEPTION_THROWN = -2
Expand Down
Loading
Loading