Skip to content

Add PyWayang load_collection using literal JSON input - #870

Open
Kaileshwar16 wants to merge 2 commits into
apache:mainfrom
Kaileshwar16:feat/pywayang-load-collection
Open

Kaileshwar16 wants to merge 2 commits into
apache:mainfrom
Kaileshwar16:feat/pywayang-load-collection

Conversation

@Kaileshwar16

Copy link
Copy Markdown
Contributor

This PR implements the load_collection() part of #869, allowing Python users to create a DataQuanta from an in-memory list or tuple.
It reuses the existing JSON execution pipeline, preserves compatibility with legacy collection plans, and validates supported values before submission.

Regression tests cover empty and nested collections, unsupported values, serialization, and real backend file output. The Python suite, Scala collection tests, and Java 17 reactor build pass. collect()

ML-specific functionality remain outside this PR’s scope.

the next part of it should be to implement DataQuanta.collect() to return execution results as a Python list

@zkaoudi
zkaoudi requested a review from juripetersen November 4, 2026 20:14
Kaileshwar16 added a commit to Kaileshwar16/wayang that referenced this pull request Nov 8, 2026
Derive serialized output connections from the selected plan's inputs so
collecting an earlier branch still works after another consumer is attached.
Add client, worker, Scala result conversion, and real backend regression
coverage, and document how to run the collection tests.

Refs apache#869; follow-up to apache#870.
assert data.operator.inputs == 0
assert data.operator.outputs == 1

with patch("pywy.core.core.requests.post") as post:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this patch doing?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the patch mocks requests.post so this unit test can inspect the generated JSON plan without needing a running REST server. Separate integration tests send real requests to the Wayang backend and verify the output files. I’ve added a comment explaining this and an assertion that exactly one request is submitted.

Comment thread python/README.md
`WAYANG_API_URL`, or `WAYANG_API_HOST` and `WAYANG_API_PORT`.

## In-memory collections

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a very long README addition for just a load collection operator.

Comment thread python/README.md Outdated
.store_textfile("file:///tmp/doubled.txt"))
```

This example uses the existing fusion of adjacent Python transformations into

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for example, this has to do with your implementation and should not be part of README

self.values = self._snapshot(collection, set(), "collection")

@staticmethod
def _snapshot(value, ancestors, path):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add support to handle NumPy values?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants